Disk integrity in QEMU

Read performance should be unaffected by using O_DSYNC. O_DIRECT will significantly reduce read performance. I think we should use O_DSYNC by default and I have sent out a patch that contains that. We will follow up with benchmarks to demonstrate this.

So to summarize, I think we should enable O_DSYNC by default to ensure that guest data integrity is not dependent on the host OS, and that practically speaking, cache=off is only useful for very specialized circumstances. Part of the patch I'll follow up with includes changes to the man page to document all of this for users.

Thoughts?

Hi,

Quoted 4 lines

Read performance should be unaffected by using O_DSYNC. O_DIRECT will significantly reduce read performance. I think we should use O_DSYNC by default and I have sent out a patch that contains that. We will follow up with benchmarks to demonstrate this.

So O_SYNC on/off is pretty much equivalent to disk write caching being on/off, right? So we could make that guest-controlled, i.e. toggeling write caching in the guest (using hdparm) toggles O_SYNC in qemu?

Hi,

Quoted 5 lines

Read performance should be unaffected by using O_DSYNC. O_DIRECT will significantly reduce read performance. I think we should use O_DSYNC by default and I have sent out a patch that contains that. We will follow up with benchmarks to demonstrate this.

So O_SYNC on/off is pretty much equivalent to disk write caching being on/off, right? So we could make that guest-controlled, i.e. toggeling write caching in the guest (using hdparm) toggles O_SYNC in qemu? This together with disk-flush command support (mapping to fsync on the host) should allow guests to go into barrier mode for better write performance without loosing data integrity.

Thoughts?

Anthony Liguori wrote:

[O_DSYNC, O_DIRECT, and 0]

Quoted 2 lines

Thoughts?

There are (at least) three usage models for qemu:

- OS development tool
- casual or client-side virtualization
- server partitioning

The last two uses are almost always in conjunction with a hypervisor.

When using qemu as an OS development tool, data integrity is not very important. On the other hand, performance and caching are, especially as the guest is likely to be restarted multiple times so the guest page cache is of limited value. For this use model the current default (write back cache) is fine.

The 'causal virtualization' use is when the user has a full native desktop, and is also running another operating system. In this case, the host page cache is likely to be larger than the guest page cache. Data integrity is important, so write-back is out of the picture. I guess for this use case O_DSYNC is preferred though O_DIRECT might not be significantly slower for long-running guests. This is because reads are unlikely to be cached and writes will not benefit much from the host pagecache.

For server partitioning, data integrity and performance are critical. The host page cache is significantly smaller than the guest page cache; if you have spare memory, give it to your guests. O_DIRECT is practically mandataed here; the host page cache does nothing except to impose an additional copy.

Given the rather small difference between O_DSYNC and O_DIRECT, I favor not adding O_DSYNC as it will add only marginal value.

Regarding choosing the default value, I think we should change the default to be safe, that is O_DIRECT. If that is regarded as too radical, the default should be O_DSYNC with options to change it to O_DIRECT or writeback. Note that some disk formats will need updating like qcow2 if they are not to have abyssal performance.

[...]
So to summarize, I think we should enable O_DSYNC by default to ensure that guest data integrity is not dependent on the host OS, and that practically speaking, cache=off is only useful for very specialized circumstances. Part of the patch I'll follow up with includes changes to the man page to document all of this for users.

Thoughts?

Quoted 6 lines

There are (at least) three usage models for qemu:

- OS development tool
- casual or client-side virtualization
- server partitioning

Quoted 4 lines

The last two uses are almost always in conjunction with a hypervisor.

When using qemu as an OS development tool, data integrity is not very important. On the other hand, performance and caching are, especially as the guest is likely to be restarted multiple times so the guest page cache is of limited value. For this use model the current default (write back cache) is fine.

It is a myth that developers dont' care about data consistency / crash
safety. I've lost countless guest VMs to corruption when my host OS crashed & its just a waste of my time. Given the choice between likely-to-corrupt and not-likely-to-corrupt, even developers will want the latter.

Absoutely agree that the default should be safe. I don't have enough knowledge to say whether O_DIRECT/O_DSYNC is best - which also implies we should choose the best setting by default, because we can't expect
users to know the tradeoffs either.

On Thu, Oct 09, 2008 at 12:00:41PM -0500, Anthony Liguori wrote:

[snip]

Quoted 5 lines

So to summarize, I think we should enable O_DSYNC by default to ensure that guest data integrity is not dependent on the host OS, and that practically speaking, cache=off is only useful for very specialized circumstances. Part of the patch I'll follow up with includes changes to the man page to document all of this for users.

Thoughts?

While I agree O_DSYNC should be the defaults, I wonder if we should keep the current behaviour available for those who want it. We can imagine the following options:
cache=off O_DIRECT
cache=read O_DSYNC (default)

cache=on 0

Anthony Liguori wrote:

[O_DSYNC, O_DIRECT, and 0]

Quoted 2 lines

Thoughts?

There are (at least) three usage models for qemu:

- OS development tool
- casual or client-side virtualization
- server partitioning

The last two uses are almost always in conjunction with a hypervisor.

When using qemu as an OS development tool, data integrity is not very important. On the other hand, performance and caching are, especially as the guest is likely to be restarted multiple times so the guest page cache is of limited value. For this use model the current default (write back cache) is fine.

The 'causal virtualization' use is when the user has a full native desktop, and is also running another operating system. In this case, the host page cache is likely to be larger than the guest page cache. Data integrity is important, so write-back is out of the picture. I guess for this use case O_DSYNC is preferred though O_DIRECT might not be significantly slower for long-running guests. This is because reads are unlikely to be cached and writes will not benefit much from the host pagecache.

For server partitioning, data integrity and performance are critical. The host page cache is significantly smaller than the guest page cache; if you have spare memory, give it to your guests.

O_DIRECT is practically mandataed here; the host page cache does nothing except to impose an additional copy.

Given the rather small difference between O_DSYNC and O_DIRECT, I favor not adding O_DSYNC as it will add only marginal value.

Regarding choosing the default value, I think we should change the default to be safe, that is O_DIRECT. If that is regarded as too radical, the default should be O_DSYNC with options to change it to O_DIRECT or writeback. Note that some disk formats will need updating like qcow2 if they are not to have abyssal performance.

But that said, I'm concerned that this is far too confusing for users. I don't think anyone is relying on disk write performance when in write-back mode simply because the guest already has a page cache so writes are already being completed instantaneously from the application's perspective.

Quoted 4 lines

For server partitioning, data integrity and performance are critical. The host page cache is significantly smaller than the guest page cache; if you have spare memory, give it to your guests.

I don't think this wisdom is bullet-proof. In the case of server partitioning, if you're designing for the future then you can assume some form of host data deduplification either through qcow deduplification, a proper content addressable storage mechanism, or file system level deduplification. It's becoming more common to see large amounts of homogeneous consolidation either because of cloud computing, virtual appliances, or just because most x86 virtualization involves Windows consolidation and there aren't that many versions of Windows.

In this case, there is an awful lot of opportunity for increasing overall system throughput by caching common data access across virtual machines.

Quoted 4 lines

O_DIRECT is practically mandataed here; the host page cache does nothing except to impose an additional copy.

Quoted 2 lines

Given the rather small difference between O_DSYNC and O_DIRECT, I favor not adding O_DSYNC as it will add only marginal value.

The difference isn't small. Our fio runs are defeating the host page cache on write so we're adjusting the working set size. But the difference in read performance between dsync and direct is many factors when the data can be cached.

Quoted 5 lines

Regarding choosing the default value, I think we should change the default to be safe, that is O_DIRECT. If that is regarded as too radical, the default should be O_DSYNC with options to change it to O_DIRECT or writeback. Note that some disk formats will need updating like qcow2 if they are not to have abyssal performance.

I think qcow2 will be okay because the only issue is image expansion and that is a relatively uncommon case that is amortized throughout the life time of the VM. So far, while there is objection to using O_DIRECT by default, I haven't seen any objection to O_DSYNC by default so as long as no one objects in the next few days, I think that's what we'll end up doing.

Quoted 8 lines

- OS development tool
- casual or client-side virtualization
- server partitioning

Absoutely agree that the default should be safe. I don't have enough knowledge to say whether O_DIRECT/O_DSYNC is best - which also implies we should choose the best setting by default, because we can't expect users to know the tradeoffs either.

The fact that there are different use models for qemu implies that the default must be chosen at some higher level than qemu code itself. It might be done using /etc/qemu or ~/.qemu, or at the management interface, but there is no best setting for qemu itself.

Quoted 10 lines

Quoted 5 lines

Absoutely agree that the default should be safe. I don't have enough knowledge to say whether O_DIRECT/O_DSYNC is best - which also implies we should choose the best setting by default, because we can't expect users to know the tradeoffs either.

The fact that there are different use models for qemu implies that the default must be chosen at some higher level than qemu code itself. It might be done using /etc/qemu or ~/.qemu, or at the management interface, but there is no best setting for qemu itself.

This suggests that the most appropriate defaults are for the users that are least likely to be using a management tool. I'd guess that the server partitioning folks are most likely to be using a management tool, so qemu defaults should be setup for casual/development use. I don't have hard data to back this up though.

Quoted 6 lines

But that said, I'm concerned that this is far too confusing for users. I don't think anyone is relying on disk write performance when in write-back mode simply because the guest already has a page cache so writes are already being completed instantaneously from the application's perspective.

This isn't entirely true. With IDE devices you don't have command queueing, so it's easy for a large write to stall subsequent reads for a relatively long time. I'm not sure how much this effects qemu, but I've definitely seen it happening on real hardware.

Paul

Quoted 11 lines · Paul Brook

Quoted 5 lines

But that said, I'm concerned that this is far too confusing for users. I don't think anyone is relying on disk write performance when in write-back mode simply because the guest already has a page cache so writes are already being completed instantaneously from the application's perspective.

This isn't entirely true. With IDE devices you don't have command queueing, so it's easy for a large write to stall subsequent reads for a relatively long time.
I'm not sure how much this effects qemu, but I've definitely seen it happening on real hardware.

I think that suggests we should have a cache=wb option and if people report slow downs with IDE, we can observe if cache=wb helps. My suspicion is that it's not going to have a practical impact because as long as the operations are asynchronous (via DMA), then you're getting native-like performance.

Quoted 7 lines

This isn't entirely true. With IDE devices you don't have command queueing, so it's easy for a large write to stall subsequent reads for a relatively long time.
I'm not sure how much this effects qemu, but I've definitely seen it happening on real hardware.

I think that suggests we should have a cache=wb option and if people report slow downs with IDE, we can observe if cache=wb helps. My suspicion is that it's not going to have a practical impact because as long as the operations are asynchronous (via DMA), then you're getting native-like performance.

My bigger concern is synchronous IO operations because then a guest VCPU is getting far less time to run and that may have a cascading effect on performance.

Quoted 15 lines · Aurelien Jarno

On Thu, Oct 09, 2008 at 12:00:41PM -0500, Anthony Liguori wrote:

[snip]

Quoted 9 lines

So to summarize, I think we should enable O_DSYNC by default to ensure that guest data integrity is not dependent on the host OS, and that practically speaking, cache=off is only useful for very specialized circumstances. Part of the patch I'll follow up with includes changes to the man page to document all of this for users.

Thoughts?

Quoted 6 lines

While I agree O_DSYNC should be the defaults, I wonder if we should keep the current behaviour available for those who want it. We can imagine the following options:
cache=off O_DIRECT
cache=read O_DSYNC (default)

Or maybe cache=off, cache=on, cache=wb. So that the default was cache=on which is write-through, or the user can choose write-back caching.

But that said, I'm concerned that this is far too confusing for users. I don't think anyone is relying on disk write performance when in write-back mode simply because the guest already has a page cache so writes are already being completed instantaneously from the application's perspective.

I think qcow2 will be okay because the only issue is image expansion and that is a relatively uncommon case that is amortized throughout the life time of the VM.

Anthony Liguori wrote:

[O_DSYNC, O_DIRECT, and 0]

Quoted 2 lines

Thoughts?

There are (at least) three usage models for qemu:

- OS development tool
- casual or client-side virtualization
- server partitioning

The last two uses are almost always in conjunction with a hypervisor.

When using qemu as an OS development tool, data integrity is not very important. On the other hand, performance and caching are, especially as the guest is likely to be restarted multiple times so the guest page cache is of limited value. For this use model the current default (write back cache) is fine.

The 'causal virtualization' use is when the user has a full native desktop, and is also running another operating system. In this case, the host page cache is likely to be larger than the guest page cache. Data integrity is important, so write-back is out of the picture. I guess for this use case O_DSYNC is preferred though O_DIRECT might not be significantly slower for long-running guests. This is because reads are unlikely to be cached and writes will not benefit much from the host pagecache.

For server partitioning, data integrity and performance are critical. The host page cache is significantly smaller than the guest page cache; if you have spare memory, give it to your guests. O_DIRECT is practically mandataed here; the host page cache does nothing except to impose an additional copy.

Given the rather small difference between O_DSYNC and O_DIRECT, I favor not adding O_DSYNC as it will add only marginal value.

Regarding choosing the default value, I think we should change the default to be safe, that is O_DIRECT. If that is regarded as too radical, the default should be O_DSYNC with options to change it to O_DIRECT or writeback. Note that some disk formats will need updating like qcow2 if they are not to have abyssal performance.

Avi Kivity wrote:

I think one of the main things to be considered is the integrity of the actual system call. The Linux manpage for open() states the following about the use of the O_DIRECT flag:

O_DIRECT (Since Linux 2.6.10)
Try to minimize cache effects of the I/O to and from this file. In general this will degrade performance, but it is useful in special situations, such as when applications do their own caching. File I/O is done directly to/from user space buffers. The I/O is synchronous, that is, at the completion of a read(2) or write(2),
data is guaranteed to have been transferred. Under Linux 2.4
transfer sizes, and the alignment of user buffer and file offset must all be multiples of the logical block size of the file system. Under Linux 2.6 alignment to 512-byte boundaries suffices.

If I focus on the sentence "The I/O is synchronous, that is, at the completion of a read(2) or write(2), data is guaranteed to have been transferred. ",

I think there a bug here. If I open a
file with the O_DIRECT flag and the host reports back to me that the transfer has completed when in fact its still in the host cache, its a bug as it violates the open()/write() call and there is no guarantee that the data will actually be written.

So I guess the real issue isn't what the default should be (although the performance team at Red Hat would vote for cache=off),

the real
issue is that we need to honor the system call from the guest. If the file is opened with O_DIRECT on the guest, then the host needs to honor that and do the same.

-mark

Quoted 18 lines · Mark Wagner

Avi Kivity wrote:

I think one of the main things to be considered is the integrity of the actual system call. The Linux manpage for open() states the following about the use of the O_DIRECT flag:

O_DIRECT (Since Linux 2.6.10)
Try to minimize cache effects of the I/O to and from this file. In general this will degrade performance, but it is useful in special situations, such as when applications do their own caching. File I/O is done directly to/from user space buffers. The I/O is synchronous, that is, at the completion of a read(2) or write(2),
data is guaranteed to have been transferred. Under Linux 2.4
transfer sizes, and the alignment of user buffer and file offset must all be multiples of the logical block size of the file system. Under Linux 2.6 alignment to 512-byte boundaries suffices.

Quoted 3 lines

If I focus on the sentence "The I/O is synchronous, that is, at the completion of a read(2) or write(2), data is guaranteed to have been transferred. ",

It's extremely important to understand what the guarantee is. The guarantee is that upon completion on write(), the data will have been reported as written by the underlying storage subsystem. This does *not* mean that the data is on disk.

If you have a normal laptop, your disk has a cache. That cache does not have a battery backup. Under normal operations, the cache is acting in write-back mode and when you do a write, the disk will report the write as completed even though it is not actually on disk. If you really care about the data being on disk, you have to either use a disk with a battery backed cache (much more expensive) or enable write-through caching (will significantly reduce performance).

In the case of KVM, even using write-back caching with the host page cache, we are still honoring the guarantee of O_DIRECT. We just have another level of caching that happens to be write-back.

Quoted 6 lines

I think there a bug here. If I open a
file with the O_DIRECT flag and the host reports back to me that the transfer has completed when in fact its still in the host cache, its a bug as it violates the open()/write() call and there is no guarantee that the data will actually be written.

This is very important, O_DIRECT does *not* guarantee that data actually resides on disk. There are many possibly places that it can be cached (in the storage controller, in the disks themselves, in a RAID controller).

Quoted 2 lines

So I guess the real issue isn't what the default should be (although the performance team at Red Hat would vote for cache=off),

The consensus so far has been that we want to still use the host page cache but use it in write-through mode. This would mean that the guest would only see data completion when the host's storage subsystem reports the write as having completed. This is not the same as cache=off but I think gives the real effect that is desired.

Do you have another argument for using cache=off?

Regards,

Anthony Liguori

Quoted 10 lines

the real
issue is that we need to honor the system call from the guest. If the file is opened with O_DIRECT on the guest, then the host needs to honor that and do the same.

-mark

Quoted 2 lines · Mark Wagner

So I guess the real issue isn't what the default should be (although the performance team at Red Hat would vote for cache=off),

The consensus so far has been that we want to still use the host page cache but use it in write-through mode. This would mean that the guest would only see data completion when the host's storage subsystem reports the write as having completed. This is not the same as cache=off but I think gives the real effect that is desired.

I think that are two distinct arguments going on here. My main concern is that I don't think that this a simple "what do we make the default cache policy be" issue. I think that regardless of the cache policy, if something in the guest requests O_DIRECT, the host must honor that and not cache the data.

Quoted 4 lines · Anthony Liguori

It's extremely important to understand what the guarantee is. The guarantee is that upon completion on write(), the data will have been reported as written by the underlying storage subsystem. This does *not* mean that the data is on disk.

I apologize if I worded it poorly, I assume that the guarantee is that the data has been sent to the storage controller and said controller sent an indication that the write has completed. This could mean multiple things likes its in the controllers cache, on the disk, etc.

I do not believe that this means that the data is still sitting in the host cache. I realize it may not yet be on a disk, but, at a minimum, I would expect that is has been sent to the storage controller. Do you consider the hosts cache to be part of the storage subsystem ?

Quoted 3 lines

In the case of KVM, even using write-back caching with the host page cache, we are still honoring the guarantee of O_DIRECT. We just have another level of caching that happens to be write-back.

I still don't get it. If I have something running on the host that I open with O_DIRECT, do you still consider it not to be a violation of the system call if that data ends up in the host cache instead of being sent to the storage controller?

Quoted 4 lines

If I focus on the sentence "The I/O is synchronous, that is, at the completion of a read(2) or write(2), data is guaranteed to have been transferred. ",

It's extremely important to understand what the guarantee is. The guarantee is that upon completion on write(), the data will have been reported as written by the underlying storage subsystem. This does *not* mean that the data is on disk.

If you have a normal laptop, your disk has a cache. That cache does not have a battery backup. Under normal operations, the cache is acting in write-back mode and when you do a write, the disk will report the write as completed even though it is not actually on disk. If you really care about the data being on disk, you have to either use a disk with a battery backed cache (much more expensive) or enable write-through caching (will significantly reduce performance).

In the case of KVM, even using write-back caching with the host page cache, we are still honoring the guarantee of O_DIRECT. We just have another level of caching that happens to be write-back.

Quoted 5 lines

I think there a bug here. If I open a
file with the O_DIRECT flag and the host reports back to me that the transfer has completed when in fact its still in the host cache, its a bug as it violates the open()/write() call and there is no guarantee that the data will actually be written.

This is very important, O_DIRECT does *not* guarantee that data actually resides on disk. There are many possibly places that it can be cached (in the storage controller, in the disks themselves, in a RAID controller).

Quoted 3 lines

So I guess the real issue isn't what the default should be (although the performance team at Red Hat would vote for cache=off),

The consensus so far has been that we want to still use the host page cache but use it in write-through mode. This would mean that the guest would only see data completion when the host's storage subsystem reports the write as having completed. This is not the same as cache=off but I think gives the real effect that is desired.

Do you have another argument for using cache=off?

I think it's safe to say the perf folks are concerned w/ data integrity first, stable/reproducible results second, and raw performance third.

So seeing data cached in host was simply not what they expected. I think write through is sufficient. However I think that uncached vs. wt will show up on the radar under reproducible results (need to tune based on cache size). And in most overcommit scenarios memory is typically more precious than cpu, it's unclear to me if the extra buffering is anything other than memory overhead. As long as it's configurable then it's comparable and benchmarking and best practices can dictate best choice.

Quoted 11 lines · Chris Wright

I think it's safe to say the perf folks are concerned w/ data integrity first, stable/reproducible results second, and raw performance third.

So seeing data cached in host was simply not what they expected. I think write through is sufficient. However I think that uncached vs. wt will show up on the radar under reproducible results (need to tune based on cache size). And in most overcommit scenarios memory is typically more precious than cpu, it's unclear to me if the extra buffering is anything other than memory overhead. As long as it's configurable then it's comparable and benchmarking and best practices can dictate best choice.

Getting good performance because we have a huge amount of free memory in the host is not a good benchmark. Under most circumstances, the free memory will be used either for more guests, or will be given to the existing guests, which can utilize it more efficiently than the host.

I can see two cases where this is not true:

- using older, 32-bit guests which cannot utilize all of the cache. I
think Windows XP is limited to 512MB of cache, and usually doesn't utilize even that. So if you have an application running on 32-bit Windows (or on 32-bit Linux with pae disabled), and a huge host, you will see a significant boost from cache=writethrough. This is a case where performance can exceed native, simply because native cannot exploit all the resources of the host.

- if cache requirements vary in time across the different guests, and
if some smart ballooning is not in place, having free memory on the host means we utilize it for whichever guest has the greatest need, so overall performance improves.

Actually this is what happens on mainline qemu with cache=off.

Either wt or uncached (so host O_DSYNC or O_DIRECT) would suffice to get it through to host's storage subsytem, and I think that's been the core of the discussion (plus defaults, etc).

I do not believe that this means that the data is still sitting in the host cache. I realize it may not yet be on a disk, but, at a minimum, I would expect that is has been sent to the storage controller. Do you consider the hosts cache to be part of the storage subsystem ?

Quoted 2 lines · Dor Laor

Actually this is what happens on mainline qemu with cache=off.

Have I understood right that cache=off on a qcow2 image only uses O_DIRECT for the leaf image, and the chain of base images don't use O_DIRECT?

Sometimes on a memory constrained host, where the (collective) guest memory is nearly as big as the host memory, I'm not sure this is what I want.

-- Jamie

Avi Kivity wrote:

Since there is ongoing effort (both by IT and developers) to keep the base images as big as possible, it guarantees that this data is best suited for caching in the host while the private leaf images will be uncached.

This way we provide good performance and caching for the shared parent images while also promising correctness.

Actually this is what happens on mainline qemu with cache=off.

Cheers,
Dor

Signature · 3 lines

To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Quoted 11 lines · Chris Wright

I think it's safe to say the perf folks are concerned w/ data integrity first, stable/reproducible results second, and raw performance third.

So seeing data cached in host was simply not what they expected. I think write through is sufficient. However I think that uncached vs. wt will show up on the radar under reproducible results (need to tune based on cache size). And in most overcommit scenarios memory is typically more precious than cpu, it's unclear to me if the extra buffering is anything other than memory overhead. As long as it's configurable then it's comparable and benchmarking and best practices can dictate best choice.

Getting good performance because we have a huge amount of free memory in the host is not a good benchmark. Under most circumstances, the free memory will be used either for more guests, or will be given to the existing guests, which can utilize it more efficiently than the host.

I can see two cases where this is not true:

- using older, 32-bit guests which cannot utilize all of the cache. I
think Windows XP is limited to 512MB of cache, and usually doesn't utilize even that. So if you have an application running on 32-bit Windows (or on 32-bit Linux with pae disabled), and a huge host, you will see a significant boost from cache=writethrough. This is a case where performance can exceed native, simply because native cannot exploit all the resources of the host.

- if cache requirements vary in time across the different guests, and
if some smart ballooning is not in place, having free memory on the host means we utilize it for whichever guest has the greatest need, so overall performance improves.

Quoted 4 lines

I do not believe that this means that the data is still sitting in the host cache. I realize it may not yet be on a disk, but, at a minimum, I would expect that is has been sent to the storage controller. Do you consider the hosts cache to be part of the storage subsystem ?

The fact that the virtualization layer has a cache is really not that unusual.

Quoted 6 lines

Getting good performance because we have a huge amount of free memory in the host is not a good benchmark. Under most circumstances, the free memory will be used either for more guests, or will be given to the existing guests, which can utilize it more efficiently than the host.

There's two arguments for O_DIRECT. The first is that you can avoid bringing in data into CPU cache. This requires zero-copy in QEMU but ignoring that, the use of the page cache doesn't necessarily prevent us from achieving this.

In the future, most systems will have a DMA offload engine. This is a pretty obvious thing to attempt to accelerate with such an engine which would prevent cache pollution.

Another possibility is to directly map the host's page cache into the guest's memory space.

The later is a bit tricky but is so much more interesting especially if you have a strong storage backend that is capable of deduplification (you get memory compaction for free).

I also have my doubts that the amount of memory saved by using O_DIRECT will have a noticable impact on performance considering that guest memory and page cache memory are entirely reclaimable.

An LRU should make the best decisions about whether memory is more valuable for the guests or for the host page cache.

LRU typically makes fairly bad decisions since it throws most of the information it has away. I recommend looking up LRU-K and similar algorithms, just to get a feel for this; it is basically the simplest possible algorithm short of random selection.

Note that Linux doesn't even have an LRU; it has to approximate since it can't sample all of the pages all of the time. With a hypervisor that uses Intel's EPT, it's even worse since we don't have an accessed bit. On silly benchmarks that just exercise the disk and touch no memory, and if you tune the host very aggresively, LRU will win on long running guests since it will eventually page out all unused guest memory (with Linux guests, it will never even page guest memory in). On real life applications I don't think there is much chance.

But would increase latency, memory bus utilization, and cpu overhead.

In the cases where the page cache buys us something (host page cache significantly larger than guest size), that's understandable. But for the other cases, why bother? Especially when many systems don't have this today.

Let me phrase this another way: is there an argument against O_DIRECT?

In a significant fraction of deployments it will be both simpler and faster.

Quoted 2 lines

Let me phrase this another way: is there an argument against O_DIRECT?

It slows down any user who frequently restarts virtual machines.

It slows down total system throughput when there are multiple virtual machines sharing a single disk. This later point is my primary concern because in the future, I expect disk sharing to be common in some form (either via common QCOW base images or via CAS).

I'd like to see a benchmark demonstrating that O_DIRECT improves overall system throughput in any scenario today. I just don't buy the cost of the extra copy today is going to be significant since the CPU cache is already polluted. I think the burden of proof is on O_DIRECT because it's quite simple to demonstrate where it hurts performance (just the time it takes to do two boots of the same image).

Quoted 3 lines

In a significant fraction of deployments it will be both simpler and faster.

I think this is speculative. Is there any performance data to back this up?

Given that we don't have a zero-copy implementation yet, it is impossible to generate real performance data.

Now I don't have data that demonstrates how bad these effects are, but I think there is sufficient arguments here to justify adding O_DIRECT. I intend to recommend O_DIRECT unless I see performance data that favours O_DSYNC on real world scenarios that take into account bandwidth, cpu utilization, and memory utilization (i.e. a 1G guest on a 32G host running fio but not top doesn't count).

The IDE write cache is not susceptible to software failures (well it is susceptible to firmware failures, but let's ignore that). It is likely to survive reset and perhaps even powerdown. The risk window is a few megabytes and tens of milliseconds long.

Quoted 3 lines

So O_SYNC on/off is pretty much equivalent to disk write caching being on/off, right? So we could make that guest-controlled, i.e. toggeling write caching in the guest (using hdparm) toggles O_SYNC in qemu?

I don't think it's correct to equate disk write caching to completing guest writes when the data has been copied to the host's page cache. The host's page cache will cache much more data for much longer than a typical disk, right?

If so, then this form of write caching is much more likely to result in fs corruption if the host crashes. In that case, all qemu users would really need to disable write caching in the guest using hdparm, which they don't need to do on bare-metal.

Quoted 5 lines · Mark Wagner

I do not believe that this means that the data is still sitting in the host cache. I realize it may not yet be on a disk, but, at a minimum, I would expect that is has been sent to the storage controller. Do you consider the hosts cache to be part of the storage subsystem ?

Yes. And the storage subsystem is often complicated like this. Consider if you had a hardware iSCSI initiator. The host just sees a SCSI disk and when the writes are issued as completed, that simply means the writes have gone to the iSCSI server. The iSCSI server may have its own cache or some deep storage multi-level cached storage subsystem.

The fact that the virtualization layer has a cache is really not that unusual.

Regards,

Anthony Liguori

If you stopped and listened to yourself, you'd see that you are making my point...

AFAIK, QEMU is neither designed nor intended to be an Enterprise Storage Array, I thought this group is designing a virtualization layer. However, the persistent argument is that since Enterprise Storage products will often acknowledge a write before the data is actually on the disk, its OK for QEMU to do the same.

If QEMU
had a similar design to Enterprise Storage with redundancy, battery backup, etc, I'd be fine with it, but you don't. QEMU is a layer that I've also thought was suppose to be small, lightweight and unobtrusive that is silently putting everyones data at risk.

The low-end iSCSI server from EqualLogic claims:
"it combines intelligence and automation with fault tolerance" "Dual, redundant controllers with a total of 4 GB battery-backed
memory"

AFAIK QEMU provides neither of these characteristics.

-mark

Quoted 1 line

The fact that the virtualization layer has a cache is really not that unusual.

Do other virtualization layers lie to the guest and indicate that the data has successfully been ACK'd by the storage subsystem when the data is actually still in the host cache?

-mark

Quoted 6 lines

Regards,

Anthony Liguori

Quoted 6 lines · Mark Wagner

If you stopped and listened to yourself, you'd see that you are making my point...

AFAIK, QEMU is neither designed nor intended to be an Enterprise Storage Array, I thought this group is designing a virtualization layer. However, the persistent argument is that since Enterprise Storage products will often acknowledge a write before the data is actually on the disk, its OK for QEMU to do the same.

I think you're a little lost in this thread. We're going to have QEMU only acknowledge writes when they complete. I've already sent out a patch. Just waiting a couple days to let everyone give their input.

Quoted 6 lines

If QEMU
had a similar design to Enterprise Storage with redundancy, battery backup, etc, I'd be fine with it, but you don't. QEMU is a layer that I've also thought was suppose to be small, lightweight and unobtrusive that is silently putting everyones data at risk.

Quoted 6 lines

The low-end iSCSI server from EqualLogic claims:
"it combines intelligence and automation with fault tolerance" "Dual, redundant controllers with a total of 4 GB battery-backed
memory"

AFAIK QEMU provides neither of these characteristics.

So if this is your only concern, we're in violent agreement. You were previously arguing that we should use O_DIRECT in the host if we're not "lying" about write completions anymore. That's what I'm opposing because the details of whether we use O_DIRECT or not have absolutely nothing to do with data integrity as long as we're using O_DSYNC.

Regards,

Anthony Liguori

Quoted 19 lines

-mark

Quoted 1 line

The fact that the virtualization layer has a cache is really not that unusual.

Do other virtualization layers lie to the guest and indicate that the data has successfully been ACK'd by the storage subsystem when the data is actually still in the host cache?

-mark

Quoted 6 lines

Regards,

Anthony Liguori

So, if your proposed default value for the cache is in effect, then O_DSYNC should provide the write-thru required by the guests use of O_DIRECT on the writes. However, if the default cache value is not used and its set to cache=on, and if the guest is using O_DIRECT or O_DSYNC, I feel there are

issues that need to be addressed.

-mark

Quoted 41 lines

Quoted 6 lines

If QEMU
had a similar design to Enterprise Storage with redundancy, battery backup, etc, I'd be fine with it, but you don't. QEMU is a layer that I've also thought was suppose to be small, lightweight and unobtrusive that is silently putting everyones data at risk.

Quoted 6 lines

The low-end iSCSI server from EqualLogic claims:
"it combines intelligence and automation with fault tolerance" "Dual, redundant controllers with a total of 4 GB battery-backed
memory"

AFAIK QEMU provides neither of these characteristics.

So if this is your only concern, we're in violent agreement. You were previously arguing that we should use O_DIRECT in the host if we're not "lying" about write completions anymore. That's what I'm opposing because the details of whether we use O_DIRECT or not have absolutely nothing to do with data integrity as long as we're using O_DSYNC.

Regards,

Anthony Liguori

Quoted 19 lines

-mark

Quoted 1 line

The fact that the virtualization layer has a cache is really not that unusual.

Do other virtualization layers lie to the guest and indicate that the data has successfully been ACK'd by the storage subsystem when the data is actually still in the host cache?

-mark

Quoted 6 lines

Regards,

Anthony Liguori

Quoted 11 lines · Anthony Liguori

Quoted 10 lines · Mark Wagner

If you stopped and listened to yourself, you'd see that you are making my point...

AFAIK, QEMU is neither designed nor intended to be an Enterprise Storage Array,
I thought this group is designing a virtualization layer. However, the persistent
argument is that since Enterprise Storage products will often acknowledge a write
before the data is actually on the disk, its OK for QEMU to do the same.

Quoted 4 lines

I think you're a little lost in this thread. We're going to have QEMU only acknowledge writes when they complete. I've already sent out a patch. Just waiting a couple days to let everyone give their input.

Actually, I'm just don't being clear enough in trying to point out that I don't think just setting a default value for "cache" goes far enough. My argument has nothing to do with the default value. It has to do with what the right thing to do is in specific situations regardless of the value of the cache setting.

My point is that if a file is opened in the guest with the O_DIRECT (or O_DSYNC) then QEMU *must* honor that regardless of whatever value the current value of "cache" is.

So, if the system admin for the host decides to set cache=on and something in the guest opens a file with O_DIRECT, I feel that it is a violation of the system call for the host to cache the write in its local cache w/o sending it immediately to the storage subsystem. It must get an ACK from the storage subsystem before it can return to the guest in order to preserve the guarantee.

So, if your proposed default value for the cache is in effect, then O_DSYNC should provide the write-thru required by the guests use of O_DIRECT on the writes. However, if the default cache value is not used and its set to cache=on, and if the guest is using O_DIRECT or O_DSYNC, I feel there are issues that need to be addressed.

Quoted 12 lines · Anthony Liguori

Quoted 10 lines · Mark Wagner

If you stopped and listened to yourself, you'd see that you are making my point...

AFAIK, QEMU is neither designed nor intended to be an Enterprise Storage Array,
I thought this group is designing a virtualization layer. However, the persistent
argument is that since Enterprise Storage products will often acknowledge a write
before the data is actually on the disk, its OK for QEMU to do the same.

Quoted 4 lines

I think you're a little lost in this thread. We're going to have QEMU only acknowledge writes when they complete. I've already sent out a patch. Just waiting a couple days to let everyone give their input.

Actually, I'm just don't being clear enough in trying to point out that I don't think just setting a default value for "cache" goes far enough. My argument has nothing to do with the default value. It has to do with what the
right thing to do is in specific situations regardless of the value of the cache setting.

My point is that if a file is opened in the guest with the O_DIRECT (or O_DSYNC)
then QEMU *must* honor that regardless of whatever value the current value of
"cache" is.

So to summarize, I think we should enable O_DSYNC by default to ensure
that guest data integrity is not dependent on the host OS, and that practically speaking, cache=off is only useful for very specialized circumstances. Part of the patch I'll follow up with includes changes
to the man page to document all of this for users.

Le jeudi 09 octobre 2008 à 12:00 -0500, Anthony Liguori a écrit :
[...]

Quoted 4 lines

So to summarize, I think we should enable O_DSYNC by default to
ensure that guest data integrity is not dependent on the host OS, and that practically speaking, cache=off is only useful for very specialized circumstances. Part of the patch I'll follow up with includes changes to the man page to document all of this for users.

perhaps I'm wrong but I think O_DSYNC (in fact O_SYNC for linux) will impact host filesystem performance, at least with ext3, because the synchronicity is done through the commit of the journal of the whole filesystem:

see fs/ext3/file.c:ext3_file_write() (I've removed the comments here) :

...

        if (file->f_flags & O_SYNC) {
 
                if (!ext3_should_journal_data(inode))
                        return ret;

                goto force_commit;
        }


        if (!IS_SYNC(inode))
                return ret;

force_commit:

        err = ext3_force_commit(inode->i_sb);
        if (err)
                return err;
        return ret;
}

Moreover, the real behavior depends on the type of the journaling system you use...

Regards,
Laurent

Read performance should be unaffected by using O_DSYNC. O_DIRECT will significantly reduce read performance. I think we should use O_DSYNC by default and I have sent out a patch that contains that. We will follow up with benchmarks to demonstrate this.

Quoted 5 lines

perhaps I'm wrong but I think O_DSYNC (in fact O_SYNC for linux) will impact host filesystem performance, at least with ext3, because the synchronicity is done through the commit of the journal of the whole filesystem:

Yes, but this is important because if the journal isn't committed, then it's possible that while the data would be on disk, the file system metadata is out of sync on disk which could result in the changes to the file being lost.

I think that you are in fact correct that the journal write is probably unnecessary overhead in a lot of scenarios but Ryan actually has some performance data that he should be posting soon that shows that in most circumstances, O_DSYNC does pretty well compared to O_DIRECT for write so I don't this is a practical concern.

When cache=on, read requests may not actually go to the disk. If a previous read request (by some application on the system) has read the same data, then it becomes a simple memcpy(). Also, the host IO scheduler may do read ahead which means that the data may be available from that.

Quoted 5 lines

When cache=on, read requests may not actually go to the disk. If a previous read request (by some application on the system) has read the same data, then it becomes a simple memcpy(). Also, the host IO scheduler may do read ahead which means that the data may be available from that.

This can be as much of a data integrity problem as asynchronous writes, if various qemu/kvm guests are accessing the same disk image with a cluster filesystem like GFS.

Quoted 6 lines · Rik van Riel

Quoted 1 line

When cache=on, read requests may not actually go to the disk. If a previous read request (by some application on the system) has read the same data, then it becomes a simple memcpy(). Also, the host IO scheduler may do read ahead which means that the data may be available from that.

This can be as much of a data integrity problem as asynchronous writes, if various qemu/kvm guests are accessing the same disk image with a cluster filesystem like GFS.

If there are multiple qemu/kvm guests accessing the same disk image in a cluster, provided the host cluster filesystem uses a fully coherent protocol, ordinary cached reads should be fine. (E.g. not NFS).

Quoted 3 lines · Anthony Liguori

Read performance should be unaffected by using O_DSYNC. O_DIRECT will significantly reduce read performance. I think we should use O_DSYNC by default and I have sent out a patch that contains that. We will follow up with benchmarks to demonstrate this.

Quoted 4 lines · Anthony Liguori

Read performance should be unaffected by using O_DSYNC. O_DIRECT will significantly reduce read performance. I think we should use O_DSYNC by default and I have sent out a patch that contains that. We will follow up with benchmarks to demonstrate this.

baremetal baseline (1g dataset):
---------------------------+-------+-------+--------------

+------------+
Test scenarios             | bandw | % CPU | ave submit   | ave compl  |
type, block size, iface    | MB/s  | usage | latency usec | latency ms |
---------------------------+-------+-------+-------------- 
+------------+
write, 16k, lvm, direct=1  | 127.7 |  12   |   11.66      |    9.48    |
write, 64k, lvm, direct=1  | 178.4 |   5   |   13.65      |   27.15    |
write, 1M,  lvm, direct=1  | 186.0 |   3   |  163.75      |  416.91    |
---------------------------+-------+-------+-------------- 
+------------+
read , 16k, lvm, direct=1  | 170.4 |  15   |   10.86      |    7.10    |
read , 64k, lvm, direct=1  | 199.2 |   5   |   12.52      |   24.31    |
read , 1M,  lvm, direct=1  | 202.0 |   3   |  133.74      |  382.67    |

---------------------------+-------+-------+--------------
+------------+

kvm write (1g dataset):
---------------------------+-------+-------+--------------

+------------+
Test scenarios             | bandw | % CPU | ave submit   | ave compl  |
block size,iface,cache,sync| MB/s  | usage | latency usec | latency ms |
---------------------------+-------+-------+-------------- 
+------------+
16k,virtio,off,none        | 135.0 |  94   |    9.1       |    8.71    |
16k,virtio,on ,none        | 184.0 | 100   |   63.69      |   63.48    |
16k,virtio,on ,O_DSYNC     | 150.0 |  35   |    6.63      |    8.31    |
---------------------------+-------+-------+-------------- 
+------------+
64k,virtio,off,none        | 169.0 |  51   |   17.10      |   28.00    |
64k,virtio,on ,none        | 189.0 |  60   |   69.42      |   24.92    |
64k,virtio,on ,O_DSYNC     | 171.0 |  48   |   18.83      |   27.72    |
---------------------------+-------+-------+-------------- 
+------------+
1M ,virtio,off,none        | 142.0 |  30   |  7176.00     |  523.00    |
1M ,virtio,on ,none        | 190.0 |  45   |  5332.63     |  392.35    |
1M ,virtio,on ,O_DSYNC     | 164.0 |  39   |  6444.48     |  471.20    |

---------------------------+-------+-------+--------------
+------------+

kvm read (1g dataset):
---------------------------+-------+-------+--------------

+------------+
Test scenarios             | bandw | % CPU | ave submit   | ave compl  |
block size,iface,cache,sync| MB/s  | usage | latency usec | latency ms |
---------------------------+-------+-------+-------------- 
+------------+
16k,virtio,off,none        | 175.0 |  40   |   22.42      |    6.71    |
16k,virtio,on ,none        | 211.0 | 147   |   59.49      |    5.54    |
16k,virtio,on ,O_DSYNC     | 212.0 | 145   |   60.45      |    5.47    |
---------------------------+-------+-------+-------------- 
+------------+
64k,virtio,off,none        | 190.0 |  64   |   16.31      |   24.92    |
64k,virtio,on ,none        | 546.0 | 161   |  111.06      |    8.54    |
64k,virtio,on ,O_DSYNC     | 520.0 | 151   |  116.66      |    8.97    |
---------------------------+-------+-------+-------------- 
+------------+
1M ,virtio,off,none        | 182.0 |  32   | 5573.44      |  407.21    |
1M ,virtio,on ,none        | 750.0 | 127   | 1344.65      |   96.42    |
1M ,virtio,on ,O_DSYNC     | 768.0 | 123   | 1289.05      |   94.25    |

---------------------------+-------+-------+--------------
+------------+

--------------------------------------------------------------------------
exporting file in ext3 filesystem as block device (1g)
--------------------------------------------------------------------------

kvm write (1g dataset):
---------------------------+-------+-------+--------------

+------------+
Test scenarios             | bandw | % CPU | ave submit   | ave compl  |
block size,iface,cache,sync| MB/s  | usage | latency usec | latency ms |
---------------------------+-------+-------+-------------- 
+------------+
16k,virtio,off,none        |  12.1 |  15   |    9.1       |    8.71    |
16k,virtio,on ,none        | 192.0 |  52   |   62.52      |    6.17    |
16k,virtio,on ,O_DSYNC     | 142.0 |  59   |   18.81      |    8.29    |
---------------------------+-------+-------+-------------- 
+------------+
64k,virtio,off,none        |  15.5 |   8   |   21.10      |  311.00    |
64k,virtio,on ,none        | 454.0 | 130   |  113.25      |   10.65    |
64k,virtio,on ,O_DSYNC     | 154.0 |  48   |   20.25      |   30.75    |
---------------------------+-------+-------+-------------- 
+------------+
1M ,virtio,off,none        |  24.7 |   5   | 41736.22     | 3020.08    |
1M ,virtio,on ,none        | 485.0 | 100   |  2052.09     |  149.81    |
1M ,virtio,on ,O_DSYNC     | 161.0 |  42   |  6268.84     |  453.84    |

---------------------------+-------+-------+--------------
+------------+

Quoted 2 lines · Anthony Liguori

Read performance should be unaffected by using O_DSYNC. O_DIRECT will significantly reduce read performance. I think we should use O_DSYNC by default and I have sent out a patch that contains that. We will follow up with benchmarks to demonstrate this.

baremetal baseline (1g dataset):
---------------------------+-------+-------+--------------+------------+

 Test scenarios             | bandw | % CPU | ave submit   | ave compl  |
 type, block size, iface    | MB/s  | usage | latency usec | latency ms |
 ---------------------------+-------+-------+--------------+------------+
 write, 16k, lvm, direct=1  | 127.7 |  12   |   11.66      |    9.48    |
 write, 64k, lvm, direct=1  | 178.4 |   5   |   13.65      |   27.15    |
 write, 1M,  lvm, direct=1  | 186.0 |   3   |  163.75      |  416.91    |
 ---------------------------+-------+-------+--------------+------------+
 read , 16k, lvm, direct=1  | 170.4 |  15   |   10.86      |    7.10    |
 read , 64k, lvm, direct=1  | 199.2 |   5   |   12.52      |   24.31    |
 read , 1M,  lvm, direct=1  | 202.0 |   3   |  133.74      |  382.67    |

---------------------------+-------+-------+--------------+------------+

kvm write (1g dataset):
---------------------------+-------+-------+--------------+------------+

 Test scenarios             | bandw | % CPU | ave submit   | ave compl  |
 block size,iface,cache,sync| MB/s  | usage | latency usec | latency ms |
 ---------------------------+-------+-------+--------------+------------+
 16k,virtio,off,none        | 135.0 |  94   |    9.1       |    8.71    |
 16k,virtio,on ,none        | 184.0 | 100   |   63.69      |   63.48    |
 16k,virtio,on ,O_DSYNC     | 150.0 |  35   |    6.63      |    8.31    |
 ---------------------------+-------+-------+--------------+------------+
 64k,virtio,off,none        | 169.0 |  51   |   17.10      |   28.00    |
 64k,virtio,on ,none        | 189.0 |  60   |   69.42      |   24.92    |
 64k,virtio,on ,O_DSYNC     | 171.0 |  48   |   18.83      |   27.72    |
 ---------------------------+-------+-------+--------------+------------+
 1M ,virtio,off,none        | 142.0 |  30   |  7176.00     |  523.00    |
 1M ,virtio,on ,none        | 190.0 |  45   |  5332.63     |  392.35    |
 1M ,virtio,on ,O_DSYNC     | 164.0 |  39   |  6444.48     |  471.20    |

---------------------------+-------+-------+--------------+------------+

kvm read (1g dataset):
---------------------------+-------+-------+--------------+------------+

 Test scenarios             | bandw | % CPU | ave submit   | ave compl  |
 block size,iface,cache,sync| MB/s  | usage | latency usec | latency ms |
 ---------------------------+-------+-------+--------------+------------+
 16k,virtio,off,none        | 175.0 |  40   |   22.42      |    6.71    |
 16k,virtio,on ,none        | 211.0 | 147   |   59.49      |    5.54    |
 16k,virtio,on ,O_DSYNC     | 212.0 | 145   |   60.45      |    5.47    |
 ---------------------------+-------+-------+--------------+------------+
 64k,virtio,off,none        | 190.0 |  64   |   16.31      |   24.92    |
 64k,virtio,on ,none        | 546.0 | 161   |  111.06      |    8.54    |
 64k,virtio,on ,O_DSYNC     | 520.0 | 151   |  116.66      |    8.97    |
 ---------------------------+-------+-------+--------------+------------+
 1M ,virtio,off,none        | 182.0 |  32   | 5573.44      |  407.21    |
 1M ,virtio,on ,none        | 750.0 | 127   | 1344.65      |   96.42    |
 1M ,virtio,on ,O_DSYNC     | 768.0 | 123   | 1289.05      |   94.25    |

---------------------------+-------+-------+--------------+------------+

--------------------------------------------------------------------------
exporting file in ext3 filesystem as block device (1g)
--------------------------------------------------------------------------

kvm write (1g dataset):
---------------------------+-------+-------+--------------+------------+

 Test scenarios             | bandw | % CPU | ave submit   | ave compl  |
 block size,iface,cache,sync| MB/s  | usage | latency usec | latency ms |
 ---------------------------+-------+-------+--------------+------------+
 16k,virtio,off,none        |  12.1 |  15   |    9.1       |    8.71    |
 16k,virtio,on ,none        | 192.0 |  52   |   62.52      |    6.17    |
 16k,virtio,on ,O_DSYNC     | 142.0 |  59   |   18.81      |    8.29    |
 ---------------------------+-------+-------+--------------+------------+
 64k,virtio,off,none        |  15.5 |   8   |   21.10      |  311.00    |
 64k,virtio,on ,none        | 454.0 | 130   |  113.25      |   10.65    |
 64k,virtio,on ,O_DSYNC     | 154.0 |  48   |   20.25      |   30.75    |
 ---------------------------+-------+-------+--------------+------------+
 1M ,virtio,off,none        |  24.7 |   5   | 41736.22     | 3020.08    |
 1M ,virtio,on ,none        | 485.0 | 100   |  2052.09     |  149.81    |
 1M ,virtio,on ,O_DSYNC     | 161.0 |  42   |  6268.84     |  453.84    |

---------------------------+-------+-------+--------------+------------+

--
Ryan Harper
Software Engineer; Linux Technology Center
IBM Corp., Austin, Tx
(512) 838-9253 T/L: 678-9253
[email protected]

Ryan Harper wrote:

Can you please post the details of the guest and host configurations.

From seeing kvm write data that is greater than that of bare metal, I would think that your test dataset is too small and not exceeding that of the host cache size.

Our previous testing has shown that once you exceed the host cache and cause the cache to flush, performance will drop to a point lower than if you didn't use the cache in the first place.

Can you repeat the tests using a data set that is 2X the size of your hosts memory and post the results for the community to see?

Quoted 9 lines · Ryan Harper

Quoted 7 lines · Anthony Liguori

Read performance should be unaffected by using O_DSYNC. O_DIRECT will
significantly reduce read performance. I think we should use O_DSYNC by
default and I have sent out a patch that contains that. We will follow
up with benchmarks to demonstrate this.

Hi Ryan,

as "cache=on" implies a factor (memory) shared by the whole system, you must take into account the size of the host memory and run some applications (several guests ?) to pollute the host cache, for instance you can run 4 guest and run bench in each of them concurrently, and you could reasonably limits the size of the host memory to 5 x the size of the guest memory.
(for instance 4 guests with 128 MB on a host with 768 MB).

as O_DSYNC implies journal commit, you should run a bench on the ext3 host file system concurrently to the bench on a guest to see the impact of the commit on each bench.

Quoted 25 lines

baremetal baseline (1g dataset):
---------------------------+-------+-------+--------------

+------------+
Test scenarios             | bandw | % CPU | ave submit   | ave  
compl  |
type, block size, iface    | MB/s  | usage | latency usec | latency  
ms |
---------------------------+-------+-------+-------------- 
+------------+
write, 16k, lvm, direct=1  | 127.7 |  12   |   11.66      |     
9.48    |
write, 64k, lvm, direct=1  | 178.4 |   5   |   13.65      |    
27.15    |
write, 1M,  lvm, direct=1  | 186.0 |   3   |  163.75      |   
416.91    |
---------------------------+-------+-------+-------------- 
+------------+
read , 16k, lvm, direct=1  | 170.4 |  15   |   10.86      |     
7.10    |
read , 64k, lvm, direct=1  | 199.2 |   5   |   12.52      |    
24.31    |
read , 1M,  lvm, direct=1  | 202.0 |   3   |  133.74      |   
382.67    |

Quoted 3 lines

---------------------------+-------+-------+--------------
+------------+

Could you recall which benchmark you use ?

Quoted 32 lines

kvm write (1g dataset):
---------------------------+-------+-------+--------------

+------------+
Test scenarios             | bandw | % CPU | ave submit   | ave  
compl  |
block size,iface,cache,sync| MB/s  | usage | latency usec | latency  
ms |
---------------------------+-------+-------+-------------- 
+------------+
16k,virtio,off,none        | 135.0 |  94   |    9.1       |     
8.71    |
16k,virtio,on ,none        | 184.0 | 100   |   63.69      |    
63.48    |
16k,virtio,on ,O_DSYNC     | 150.0 |  35   |    6.63      |     
8.31    |
---------------------------+-------+-------+-------------- 
+------------+
64k,virtio,off,none        | 169.0 |  51   |   17.10      |    
28.00    |
64k,virtio,on ,none        | 189.0 |  60   |   69.42      |    
24.92    |
64k,virtio,on ,O_DSYNC     | 171.0 |  48   |   18.83      |    
27.72    |
---------------------------+-------+-------+-------------- 
+------------+
1M ,virtio,off,none        | 142.0 |  30   |  7176.00     |   
523.00    |
1M ,virtio,on ,none        | 190.0 |  45   |  5332.63     |   
392.35    |
1M ,virtio,on ,O_DSYNC     | 164.0 |  39   |  6444.48     |   
471.20    |

Quoted 2 lines

---------------------------+-------+-------+--------------
+------------+

According to the semantic, I don't understand how O_DSYNC can be better than cache=off in this case...

Quoted 33 lines

kvm read (1g dataset):
---------------------------+-------+-------+--------------

+------------+
Test scenarios             | bandw | % CPU | ave submit   | ave  
compl  |
block size,iface,cache,sync| MB/s  | usage | latency usec | latency  
ms |
---------------------------+-------+-------+-------------- 
+------------+
16k,virtio,off,none        | 175.0 |  40   |   22.42      |     
6.71    |
16k,virtio,on ,none        | 211.0 | 147   |   59.49      |     
5.54    |
16k,virtio,on ,O_DSYNC     | 212.0 | 145   |   60.45      |     
5.47    |
---------------------------+-------+-------+-------------- 
+------------+
64k,virtio,off,none        | 190.0 |  64   |   16.31      |    
24.92    |
64k,virtio,on ,none        | 546.0 | 161   |  111.06      |     
8.54    |
64k,virtio,on ,O_DSYNC     | 520.0 | 151   |  116.66      |     
8.97    |
---------------------------+-------+-------+-------------- 
+------------+
1M ,virtio,off,none        | 182.0 |  32   | 5573.44      |   
407.21    |
1M ,virtio,on ,none        | 750.0 | 127   | 1344.65      |    
96.42    |
1M ,virtio,on ,O_DSYNC     | 768.0 | 123   | 1289.05      |    
94.25    |

Quoted 2 lines

---------------------------+-------+-------+--------------
+------------+

OK, but in this case the size of the cache for "cache=off" is the size of the guest cache whereas in the other cases the size of the cache is the size of the guest cache + the size of the host cache, this is not fair...

Quoted 37 lines

--------------------------------------------------------------------------
exporting file in ext3 filesystem as block device (1g)
--------------------------------------------------------------------------

kvm write (1g dataset):
---------------------------+-------+-------+--------------

+------------+
Test scenarios             | bandw | % CPU | ave submit   | ave  
compl  |
block size,iface,cache,sync| MB/s  | usage | latency usec | latency  
ms |
---------------------------+-------+-------+-------------- 
+------------+
16k,virtio,off,none        |  12.1 |  15   |    9.1       |     
8.71    |
16k,virtio,on ,none        | 192.0 |  52   |   62.52      |     
6.17    |
16k,virtio,on ,O_DSYNC     | 142.0 |  59   |   18.81      |     
8.29    |
---------------------------+-------+-------+-------------- 
+------------+
64k,virtio,off,none        |  15.5 |   8   |   21.10      |   
311.00    |
64k,virtio,on ,none        | 454.0 | 130   |  113.25      |    
10.65    |
64k,virtio,on ,O_DSYNC     | 154.0 |  48   |   20.25      |    
30.75    |
---------------------------+-------+-------+-------------- 
+------------+
1M ,virtio,off,none        |  24.7 |   5   | 41736.22     |  
3020.08    |
1M ,virtio,on ,none        | 485.0 | 100   |  2052.09     |   
149.81    |
1M ,virtio,on ,O_DSYNC     | 161.0 |  42   |  6268.84     |   
453.84    |

Quoted 2 lines

---------------------------+-------+-------+--------------
+------------+

What file type do you use (qcow2, raw ?).

Quoted 15 lines · Laurent Vivier

Quoted 9 lines · Ryan Harper

Quoted 7 lines · Anthony Liguori

Read performance should be unaffected by using O_DSYNC. O_DIRECT will
significantly reduce read performance. I think we should use O_DSYNC by
default and I have sent out a patch that contains that. We will follow
up with benchmarks to demonstrate this.

Hi Ryan,

Quoted 7 lines

as "cache=on" implies a factor (memory) shared by the whole system, you must take into account the size of the host memory and run some applications (several guests ?) to pollute the host cache, for instance you can run 4 guest and run bench in each of them concurrently, and you could reasonably limits the size of the host memory to 5 x the size of the guest memory.
(for instance 4 guests with 128 MB on a host with 768 MB).

I'm not following you here, the only assumption I see is that we have 1g of host mem free for caching the write.

Quoted 5 lines

as O_DSYNC implies journal commit, you should run a bench on the ext3 host file system concurrently to the bench on a guest to see the impact of the commit on each bench.

I understand the goal here, but what sort of host ext3 journaling load is appropriate. Additionally, when we're exporting block devices, I don't believe the ext3 journal is an issue.

Quoted 31 lines

Quoted 30 lines

baremetal baseline (1g dataset):
---------------------------+-------+-------+--------------

+------------+
Test scenarios             | bandw | % CPU | ave submit   | ave
compl  |
type, block size, iface    | MB/s  | usage | latency usec | latency
ms |
---------------------------+-------+-------+--------------
+------------+
write, 16k, lvm, direct=1  | 127.7 |  12   |   11.66      |
9.48    |
write, 64k, lvm, direct=1  | 178.4 |   5   |   13.65      |
27.15    |
write, 1M,  lvm, direct=1  | 186.0 |   3   |  163.75      |
416.91    |
---------------------------+-------+-------+--------------
+------------+
read , 16k, lvm, direct=1  | 170.4 |  15   |   10.86      |
7.10    |
read , 64k, lvm, direct=1  | 199.2 |   5   |   12.52      |
24.31    |
read , 1M,  lvm, direct=1  | 202.0 |   3   |  133.74      |
382.67    |

---------------------------+-------+-------+--------------
+------------+

Quoted 1 line

Could you recall which benchmark you use ?

yeah:

fio --name=guestrun --filename=/dev/vda --rw=write --bs=${SIZE} --ioengine=libaio --direct=1 --norandommap --numjobs=1 -- group_reporting
--thread --size=1g --write_lat_log --write_bw_log --iodepth=74

Quoted 35 lines

Quoted 34 lines

kvm write (1g dataset):
---------------------------+-------+-------+--------------

+------------+
Test scenarios             | bandw | % CPU | ave submit   | ave
compl  |
block size,iface,cache,sync| MB/s  | usage | latency usec | latency
ms |
---------------------------+-------+-------+--------------
+------------+
16k,virtio,off,none        | 135.0 |  94   |    9.1       |
8.71    |
16k,virtio,on ,none        | 184.0 | 100   |   63.69      |
63.48    |
16k,virtio,on ,O_DSYNC     | 150.0 |  35   |    6.63      |
8.31    |
---------------------------+-------+-------+--------------
+------------+
64k,virtio,off,none        | 169.0 |  51   |   17.10      |
28.00    |
64k,virtio,on ,none        | 189.0 |  60   |   69.42      |
24.92    |
64k,virtio,on ,O_DSYNC     | 171.0 |  48   |   18.83      |
27.72    |
---------------------------+-------+-------+--------------
+------------+
1M ,virtio,off,none        | 142.0 |  30   |  7176.00     |
523.00    |
1M ,virtio,on ,none        | 190.0 |  45   |  5332.63     |
392.35    |
1M ,virtio,on ,O_DSYNC     | 164.0 |  39   |  6444.48     |
471.20    |

---------------------------+-------+-------+--------------
+------------+

Quoted 2 lines

According to the semantic, I don't understand how O_DSYNC can be better than cache=off in this case...

I don't have a good answer either, but O_DIRECT and O_DSYNC are different paths through the kernel. This deserves a better reply, but I don't have one off the top of my head.

Quoted 38 lines

Quoted 37 lines

kvm read (1g dataset):
---------------------------+-------+-------+--------------

+------------+
Test scenarios             | bandw | % CPU | ave submit   | ave
compl  |
block size,iface,cache,sync| MB/s  | usage | latency usec | latency
ms |
---------------------------+-------+-------+--------------
+------------+
16k,virtio,off,none        | 175.0 |  40   |   22.42      |
6.71    |
16k,virtio,on ,none        | 211.0 | 147   |   59.49      |
5.54    |
16k,virtio,on ,O_DSYNC     | 212.0 | 145   |   60.45      |
5.47    |
---------------------------+-------+-------+--------------
+------------+
64k,virtio,off,none        | 190.0 |  64   |   16.31      |
24.92    |
64k,virtio,on ,none        | 546.0 | 161   |  111.06      |
8.54    |
64k,virtio,on ,O_DSYNC     | 520.0 | 151   |  116.66      |
8.97    |
---------------------------+-------+-------+--------------
+------------+
1M ,virtio,off,none        | 182.0 |  32   | 5573.44      |
407.21    |
1M ,virtio,on ,none        | 750.0 | 127   | 1344.65      |
96.42    |
1M ,virtio,on ,O_DSYNC     | 768.0 | 123   | 1289.05      |
94.25    |

---------------------------+-------+-------+--------------
+------------+

Quoted 4 lines

OK, but in this case the size of the cache for "cache=off" is the size of the guest cache whereas in the other cases the size of the cache is the size of the guest cache + the size of the host cache, this is not fair...

it isn't supposed to be fair, cache=off is O_DIRECT, we're reading from the device, we *want* to be able to lean on the host cache to read the data, pay once and benefit in other guests if possible.

Quoted 41 lines

Quoted 40 lines

--------------------------------------------------------------------------
exporting file in ext3 filesystem as block device (1g)
--------------------------------------------------------------------------

kvm write (1g dataset):
---------------------------+-------+-------+--------------

+------------+
Test scenarios             | bandw | % CPU | ave submit   | ave
compl  |
block size,iface,cache,sync| MB/s  | usage | latency usec | latency
ms |
---------------------------+-------+-------+--------------
+------------+
16k,virtio,off,none        |  12.1 |  15   |    9.1       |
8.71    |
16k,virtio,on ,none        | 192.0 |  52   |   62.52      |
6.17    |
16k,virtio,on ,O_DSYNC     | 142.0 |  59   |   18.81      |
8.29    |
---------------------------+-------+-------+--------------
+------------+
64k,virtio,off,none        |  15.5 |   8   |   21.10      |
311.00    |
64k,virtio,on ,none        | 454.0 | 130   |  113.25      |
10.65    |
64k,virtio,on ,O_DSYNC     | 154.0 |  48   |   20.25      |
30.75    |
---------------------------+-------+-------+--------------
+------------+
1M ,virtio,off,none        |  24.7 |   5   | 41736.22     |
3020.08    |
1M ,virtio,on ,none        | 485.0 | 100   |  2052.09     |
149.81    |
1M ,virtio,on ,O_DSYNC     | 161.0 |  42   |  6268.84     |
453.84    |

---------------------------+-------+-------+--------------
+------------+

Quoted 1 line

What file type do you use (qcow2, raw ?).

Raw.

Quoted 9 lines

Quoted 8 lines

as "cache=on" implies a factor (memory) shared by the whole system, you must take into account the size of the host memory and run some applications (several guests ?) to pollute the host cache, for instance you can run 4 guest and run bench in each of them concurrently, and you could reasonably limits the size of the host memory to 5 x the size of the guest memory.
(for instance 4 guests with 128 MB on a host with 768 MB).

Quoted 3 lines

I'm not following you here, the only assumption I see is that we have 1g
of host mem free for caching the write.

Is this a realistic use case ?

Quoted 8 lines

Quoted 4 lines

as O_DSYNC implies journal commit, you should run a bench on the ext3 host file system concurrently to the bench on a guest to see the impact of the commit on each bench.

I understand the goal here, but what sort of host ext3 journaling load is appropriate. Additionally, when we're exporting block devices, I don't believe the ext3 journal is an issue.

Yes, it's a comment for the last test case.
I think you can run the same benchmark as you do in the guest.

Quoted 6 lines

Quoted 2 lines

According to the semantic, I don't understand how O_DSYNC can be better than cache=off in this case...

I don't have a good answer either, but O_DIRECT and O_DSYNC are different paths through the kernel. This deserves a better reply, but I don't have one off the top of my head.

The O_DIRECT kernel path should be more "direct" than the O_DSYNC one. Perhaps a oprofile could help to understand ?
What it is strange also is the CPU usage with cache=off. It should be lower than others, perhaps an alignment issue ?
due to the LVM ?

Quoted 8 lines

Quoted 7 lines

OK, but in this case the size of the cache for "cache=off" is the size
of the guest cache whereas in the other cases the size of the cache is
the size of the guest cache + the size of the host cache, this is not fair...

Quoted 4 lines

it isn't supposed to be fair, cache=off is O_DIRECT, we're reading from
the device, we *want* to be able to lean on the host cache to read the data, pay once and benefit in other guests if possible.

OK, but if you want to follow this way I think you must run several guests concurrently to see how the host cache help each of them. If you want I can try this tomorrow ? The O_DSYNC patch is the one posted to the mailing-list ?

And moreover, you should run an endurance test to see how the cache evolves.

Quoted 3 lines

If there are multiple qemu/kvm guests accessing the same disk image in a cluster, provided the host cluster filesystem uses a fully coherent protocol, ordinary cached reads should be fine. (E.g. not NFS).

The problem is when the synchronization only happens in the guests, which is a legitimate and common configuration.

Ie. the hosts just pass through the IO and the guests run a GFS cluster.

Caching either reads or writes at the host level causes problems.

Is there a reason why you wouldn't use, say, GFS on the host (because it claims to be coherent)? Does performance suck relative to O_DIRECT over NFS?

Quoted 1 line · Laurent Vivier

What file type do you use (qcow2, raw ?).

Raw.

Quoted 3 lines · Ryan Harper

Quoted 1 line · Laurent Vivier

What file type do you use (qcow2, raw ?).

Quoted 1 line

Raw.

I guess the image is preallocated? What about sparse files (or qcow2, anything that grows), do have numbers on those? In the past, I experienced O_DIRECT to be horribly slow on them.

Well, looking at your numbers, they _are_ quite bad, so maybe it actually was sparse. Then the preallocated case would be interesting.

Quoted 2 lines · Avi Kivity

Given that we don't have a zero-copy implementation yet, it is impossible to generate real performance data.

Which means that it's premature to suggest switching the default to O_DIRECT since it's not going to help right now. It can be revisited once we can do zero copy but again, I think it should be driven by actual performance data. My main point is that switching to O_DIRECT right now is only going to hurt performance for some users, and most likely help no one.

Quoted 7 lines

Now I don't have data that demonstrates how bad these effects are, but I think there is sufficient arguments here to justify adding O_DIRECT. I intend to recommend O_DIRECT unless I see performance data that favours O_DSYNC on real world scenarios that take into account bandwidth, cpu utilization, and memory utilization (i.e. a 1G guest on a 32G host running fio but not top doesn't count).

So you intend on recommending something that you don't think is going to improve performance today and you know in certain scenarios is going to decrease performance? That doesn't seem right :-)

I'm certainly open to changing the default once we get to a point where there's a demonstrable performance improvement from O_DIRECT but since I don't think it's a given that there will be, switching now seems like a premature optimization which has the side effect of reducing the performance of certain users. That seems like a Bad Thing to me.

I don't think O_DIRECT should be qemu's default, since anyone using qemu directly is likely a "causal virtualization" user. Management systems like ovirt should definitely default to O_DIRECT (really, they shouldn't even offer caching).

Quoted 5 lines · Avi Kivity

I don't think O_DIRECT should be qemu's default, since anyone using qemu directly is likely a "causal virtualization" user. Management systems like ovirt should definitely default to O_DIRECT (really, they shouldn't even offer caching).

ovirt isn't a good example because the default storage model is iSCSI. Since you aren't preserving zero-copy, I doubt that you'll see any advantage to using O_DIRECT (I suspect the code paths aren't even different).

fio --name=guestrun --filename=/dev/vda --rw=write --bs=${SIZE} --ioengine=libaio --direct=1 --norandommap --numjobs=1 --group_reporting --thread --size=1g --write_lat_log --write_bw_log --iodepth=74

Quoted 5 lines

OK, but in this case the size of the cache for "cache=off" is the size of the guest cache whereas in the other cases the size of the cache is the size of the guest cache + the size of the host cache, this is not fair...

it isn't supposed to be fair, cache=off is O_DIRECT, we're reading from the device, we *want* to be able to lean on the host cache to read the data, pay once and benefit in other guests if possible.

With 16k writes I think we hit a pathological case with the particular storage backend we're using since it has many disks and the volume is striped. Also the results a bit different when going through a file system verses a LVM partition (the later being the first data set). Presumably, this is because even with no flags, writes happen synchronously to a LVM partition.

Also, cache=off seems to do pretty terribly when operating on an ext3 file. I suspect this has to do with how ext3 implements O_DIRECT.

However, the data demonstrates pretty nicely that O_DSYNC gives you native write speed, but accelerated read speed which I think we agree is the desirable behavior. cache=off never seems to outperform cache=wt which is another good argument for it being the default over cache=off.

Quoted 4 lines

From seeing kvm write data that is greater than that of bare metal, I would think that your test dataset is too small and not exceeding that of the host cache size.

The size was chosen so it would fit in to demonstrate the crazy #'s seen on cached writes without O_DSYNC.

But when using O_DIRECT you actuality make the pages not swappable at all...
or am i wrong?

However, should the effect of the guest turning off the IDE disk write cache perhaps be identical to the guest issuing IDE cache flush commands following every IDE write?

This could mean the host calling fdatasync, or fsync, or using O_DSYNC, or O_DIRECT - whatever the host does for IDE flush cache.

What this means _exactly_ for data integrity is outside of qemu's control and is a user & host configuration issue. But qemu could provide consistency at least.

Quoted 4 lines · Anthony Liguori

Quoted 2 lines · Avi Kivity

Given that we don't have a zero-copy implementation yet, it is impossible to generate real performance data.

Quoted 6 lines

Which means that it's premature to suggest switching the default to O_DIRECT since it's not going to help right now. It can be revisited once we can do zero copy but again, I think it should be driven by actual performance data. My main point is that switching to O_DIRECT right now is only going to hurt performance for some users, and most likely help no one.

I am assuming that we will provide true O_DIRECT support soon.

You get correctness by using O_DSYNC. cache=off should disable the use of the page cache everywhere.

=64k to reach platter speed.

Quoted 2 lines

I am assuming that we will provide true O_DIRECT support soon.

If you remember, I tried to introduce zero copy when I wrote the "cache=off" patch:

http://thread.gmane.org/gmane.comp.emulators.qemu/22148/focus=22149

but it was not correct (see Fabrice comment).

Hi,

I've made a benchmark using a database:
mysql and sysbench in OLTP mode.

cache=off seems to be the best choice in this case...

mysql database
http://sysbench.sourceforge.net

sysbench --test=oltp

200,000 requests on 2,000,000 rows table.

                 | total time | per-request stat (ms) |
                 |  (seconds) |  min  |  avg  |  max  |

-----------------+------------+-------+-------+-------+
baremetal | 208.6237 | 2.5 | 16.7 | 942.6 |
-----------------+------------+-------+-------+-------+
cache=on | 642.2962 | 2.5 | 51.4 | 326.9 |
-----------------+------------+-------+-------+-------+
cache=on,O_DSYNC | 646.6570 | 2.7 | 51.7 | 347.0 |
-----------------+------------+-------+-------+-------+
cache=off | 635.4424 | 2.9 | 50.8 | 399.5 |
-----------------+------------+-------+-------+-------+

Laurent

Quoted 7 lines · Laurent Vivier

Hi,

I've made a benchmark using a database:
mysql and sysbench in OLTP mode.

cache=off seems to be the best choice in this case...

It would be interesting for you to run the same workload under KVM.

Quoted 9 lines

mysql database
http://sysbench.sourceforge.net

sysbench --test=oltp

200,000 requests on 2,000,000 rows table.

                 | total time | per-request stat (ms) |
                 |  (seconds) |  min  |  avg  |  max  |

Quoted 10 lines

-----------------+------------+-------+-------+-------+
baremetal | 208.6237 | 2.5 | 16.7 | 942.6 |
-----------------+------------+-------+-------+-------+
cache=on | 642.2962 | 2.5 | 51.4 | 326.9 |
-----------------+------------+-------+-------+-------+
cache=on,O_DSYNC | 646.6570 | 2.7 | 51.7 | 347.0 |
-----------------+------------+-------+-------+-------+
cache=off | 635.4424 | 2.9 | 50.8 | 399.5 |
-----------------+------------+-------+-------+-------+

Because you're talking about 1/3% of native performance. This means that you may be dominated by things like CPU overhead verses actual IO throughput.

Quoted 13 lines

                 | total time | per-request stat (ms) |
                 |  (seconds) |  min  |  avg  |  max  |

-----------------+------------+-------+-------+-------+
baremetal | 208.6237 | 2.5 | 16.7 | 942.6 |
-----------------+------------+-------+-------+-------+
cache=on | 642.2962 | 2.5 | 51.4 | 326.9 |
-----------------+------------+-------+-------+-------+
cache=on,O_DSYNC | 646.6570 | 2.7 | 51.7 | 347.0 |
-----------------+------------+-------+-------+-------+
cache=off | 635.4424 | 2.9 | 50.8 | 399.5 |
-----------------+------------+-------+-------+-------+

Because you're talking about 1/3% of native performance. This means that you may be dominated by things like CPU overhead verses actual IO throughput.

Quoted 14 lines · Anthony Liguori

Quoted 7 lines

This isn't entirely true. With IDE devices you don't have command queueing, so it's easy for a large write to stall subsequent reads for a relatively long time.
I'm not sure how much this effects qemu, but I've definitely seen it happening on real hardware.

I think that suggests we should have a cache=wb option and if people report slow downs with IDE, we can observe if cache=wb helps. My suspicion is that it's not going to have a practical impact because as long as the operations are asynchronous (via DMA), then you're getting native-like performance.

Quoted 3 lines

My bigger concern is synchronous IO operations because then a guest VCPU is getting far less time to run and that may have a cascading effect on performance.

IDE is limited to 256 sectors per transaction, or 128KB. If a sync transaction takes 5 ms, then your write rate is limited to 25 MB/sec. It's much worse if you're allocating qcow2 data, so each transaction is several sync writes.

Quoted 14 lines · Anthony Liguori

Quoted 13 lines

                 | total time | per-request stat (ms) |
                 |  (seconds) |  min  |  avg  |  max  |

-----------------+------------+-------+-------+-------+
baremetal | 208.6237 | 2.5 | 16.7 | 942.6 |
-----------------+------------+-------+-------+-------+
cache=on | 642.2962 | 2.5 | 51.4 | 326.9 |
-----------------+------------+-------+-------+-------+
cache=on,O_DSYNC | 646.6570 | 2.7 | 51.7 | 347.0 |
-----------------+------------+-------+-------+-------+
cache=off | 635.4424 | 2.9 | 50.8 | 399.5 |
-----------------+------------+-------+-------+-------+

Quoted 3 lines

Because you're talking about 1/3% of native performance. This means that you may be dominated by things like CPU overhead verses actual IO throughput.

I don't know mysql well, but perhaps it sizes its internal cache to system memory size, so baremetal has 4x the amount of cache.

If mysql uses mmap to access its data files, then it automatically scales with system memory.

Quoted 8 lines

If you have a normal laptop, your disk has a cache. That cache does not have a battery backup. Under normal operations, the cache is acting in write-back mode and when you do a write, the disk will report the write as completed even though it is not actually on disk. If you really care about the data being on disk, you have to either use a disk with a battery backed cache (much more expensive) or enable write-through caching (will significantly reduce performance).

I think that with SATA NCQ, this is no longer true. The drive will report the write complete when it is on disk, and utilize multiple outstanding requests to get coalescing and reordering. Not sure about

this, though -- some drives may still be lying.

Quoted 3 lines · Avi Kivity

Quoted 3 lines

If you have a normal laptop, your disk has a cache. That cache does not have a battery backup. Under normal operations, the cache is acting in write-back mode and when you do a write, the disk will report the write as completed even though it is not actually on disk. If you really care about the data being on disk, you have to either use a disk with a battery backed cache (much more expensive) or enable write-through caching (will significantly reduce performance).

Quoted 2 lines

I think that with SATA NCQ, this is no longer true. The drive will report the write complete when it is on disk, and utilize multiple outstanding requests to get coalescing and reordering. Not sure about

It is still very true. Go buy any consumer drive on the market and check the write cache settings - hint, it's definitely shipped with write back caching. So while the drive may have NCQ and Linux will use it, the write cache is still using write back unless you explicitly change it.

Quoted 4 lines

IDE is limited to 256 sectors per transaction, or 128KB. If a sync
transaction takes 5 ms, then your write rate is limited to 25 MB/sec. It's much worse if you're allocating qcow2 data, so each transaction is several sync writes.

No it isn't, even most IDE drives support lba48 which raises that limit to 64K sectors, or 32MB.

Quoted 16 lines · Jens Axboe

Quoted 9 lines · Avi Kivity

Quoted 9 lines

If you have a normal laptop, your disk has a cache. That cache does not have a battery backup. Under normal operations, the cache is acting in write-back mode and when you do a write, the disk will report the write as completed even though it is not actually on disk. If you really care about the data being on disk, you have to either use a disk with a battery backed cache (much more expensive) or enable write-through caching (will significantly reduce performance).

Quoted 4 lines

I think that with SATA NCQ, this is no longer true. The drive will report the write complete when it is on disk, and utilize multiple outstanding requests to get coalescing and reordering. Not sure about

Quoted 6 lines

It is still very true. Go buy any consumer drive on the market and check the write cache settings - hint, it's definitely shipped with write back caching. So while the drive may have NCQ and Linux will use it, the write cache is still using write back unless you explicitly change it.

Sounds like a bug. Shouldn't Linux disable the write cache unless the user explicitly enables it, if NCQ is available? NCQ should provide acceptable throughput even without the write cache.

Quoted 4 lines

Sounds like a bug. Shouldn't Linux disable the write cache unless the user explicitly enables it, if NCQ is available? NCQ should provide acceptable throughput even without the write cache.

How can it be a bug?

Changing the cache policy of a drive would be a
policy decision in the kernel,

that is never the right thing to do.
There's no such thing as 'acceptable throughput',

manufacturers and
customers usually just want the go faster stripes and data consistency is second.

Additionally, write back caching is perfectly safe, if used with a barrier enabled file system in Linux.

Also note that most users will not have deep queuing for most things. To get good random write performance with write through caching and NCQ, you naturally need to be able to fill the drive queue most of the time. Most desktop workloads don't come close to that, so the user will definitely see it as slower.

Quoted 4 lines

Sounds like a bug. Shouldn't Linux disable the write cache unless the user explicitly enables it, if NCQ is available? NCQ should provide acceptable throughput even without the write cache.

How can it be a bug?

Changing the cache policy of a drive would be a
policy decision in the kernel,

that is never the right thing to do.
There's no such thing as 'acceptable throughput',

manufacturers and
customers usually just want the go faster stripes and data consistency is second.

Additionally, write back caching is perfectly safe, if used with a barrier enabled file system in Linux.

Also note that most users will not have deep queuing for most things. To get good random write performance with write through caching and NCQ, you naturally need to be able to fill the drive queue most of the time. Most desktop workloads don't come close to that, so the user will definitely see it as slower.

Quoted 5 lines · Jens Axboe

Quoted 4 lines

Sounds like a bug. Shouldn't Linux disable the write cache unless the user explicitly enables it, if NCQ is available? NCQ should provide acceptable throughput even without the write cache.

Quoted 1 line

How can it be a bug?

If it puts my data at risk, it's a bug. I can understand it for IDE, but not for SATA with NCQ.

Quoted 2 lines

Changing the cache policy of a drive would be a
policy decision in the kernel,

If you don't want this in the kernel, then the system as a whole should default to being safe. Though in this case I think it is worthwhile to do this in the kernel.

Quoted 2 lines

that is never the right thing to do.
There's no such thing as 'acceptable throughput',

I meant that performance is not completely destroyed. How can you even

compare data safety to some percent of performance?

Quoted 3 lines

manufacturers and
customers usually just want the go faster stripes and data consistency is second.

What is the performance impact of disabling the write cache, given enough queue depth?

Quoted 3 lines

Additionally, write back caching is perfectly safe, if used with a barrier enabled file system in Linux.

Not all Linux filesystems are barrier enabled, AFAIK. Further, barriers don't help with O_DIRECT (right?).

I shouldn't need a disk array to run a database.

Quoted 6 lines

Also note that most users will not have deep queuing for most things. To get good random write performance with write through caching and NCQ, you naturally need to be able to fill the drive queue most of the time. Most desktop workloads don't come close to that, so the user will definitely see it as slower.

Most desktop workloads use writeback cache, so write performance is not critical.

However I'd hate to see my data destroyed by a power failure, and today's large caches can hold a bunch of data.

Quoted 10 lines · Avi Kivity

Quoted 5 lines · Jens Axboe

Quoted 5 lines

Sounds like a bug. Shouldn't Linux disable the write cache unless the user explicitly enables it, if NCQ is available? NCQ should provide acceptable throughput even without the write cache.

Quoted 5 lines

Quoted 2 lines

How can it be a bug?

If it puts my data at risk, it's a bug. I can understand it for IDE, but not for SATA with NCQ.

Then YOU turn it off. Other people would consider the lousy performance to be the bigger problem. See policy :-)

Quoted 7 lines

Quoted 3 lines

Changing the cache policy of a drive would be a policy decision in the kernel,

If you don't want this in the kernel, then the system as a whole should default to being safe. Though in this case I think it is worthwhile to do this in the kernel.

Doesn't matter how you turn this, it's still a policy decision. Leave it to the user. It's not exactly a new turn of events, commodity drives have shipped with write caching on forever. What if the drive has a battery backing?

What if the user has an UPS?

Quoted 5 lines

Quoted 3 lines

that is never the right thing to do.
There's no such thing as 'acceptable throughput',

I meant that performance is not completely destroyed. How can you even

How do you know it's not destroyed? Depending on your workload, it may very well be dropping your throughput by orders of magnitude.

Quoted 2 lines

compare data safety to some percent of performance?

I'm not, what I'm saying is that different people will have different opponions on what is most important. Do note that the window of corruption is really small and requires powerloss to trigger. So for most desktop users, the tradeoff is actually sane.

Quoted 4 lines

Quoted 4 lines

Additionally, write back caching is perfectly safe, if used with a barrier enabled file system in Linux.

Quoted 3 lines

Not all Linux filesystems are barrier enabled, AFAIK. Further, barriers don't help with O_DIRECT (right?).

O_DIRECT should just use FUA writes, there are safe with write back caching. I'm actually testing such a change just to gauge the performance impact.

Quoted 3 lines

I shouldn't need a disk array to run a database.

You are free to turn off write back caching!

Quoted 4 lines

Most desktop workloads use writeback cache, so write performance is not critical.

Ehm, how do you reach that conclusion based on that statement?

Quoted 3 lines

However I'd hate to see my data destroyed by a power failure, and today's large caches can hold a bunch of data.

Then you use barriers or turn write back caching off, simple as that.

Quoted 12 lines · Jens Axboe

Quoted 10 lines · Avi Kivity

Quoted 5 lines · Jens Axboe

Quoted 5 lines

Sounds like a bug. Shouldn't Linux disable the write cache unless the user explicitly enables it, if NCQ is available? NCQ should provide acceptable throughput even without the write cache.

Quoted 10 lines

Quoted 5 lines

Quoted 2 lines

How can it be a bug?

If it puts my data at risk, it's a bug. I can understand it for IDE, but not for SATA with NCQ.

Then YOU turn it off. Other people would consider the lousy performance to be the bigger problem. See policy :-)

If I get lousy performance, I can turn on the write cache and ignore the risk of data loss. If I lose my data, I can't turn off the write cache and get my data back.

(it seems I can't turn off the write cache even without losing my data:

[avi@firebolt ~]$ sudo sdparm --set=WCE=0 /dev/sd[ab]

    /dev/sda: ATA       WDC WD3200YS-01P  21.0
change_mode_page: failed setting page: Caching (SBC)
    /dev/sdb: ATA       WDC WD3200YS-01P  21.0

change_mode_page: failed setting page: Caching (SBC)

Quoted 8 lines

Quoted 7 lines

Quoted 3 lines

Changing the cache policy of a drive would be a policy decision in the kernel,

If you don't want this in the kernel, then the system as a whole should default to being safe. Though in this case I think it is worthwhile to do this in the kernel.

Quoted 4 lines

Doesn't matter how you turn this, it's still a policy decision. Leave it to the user. It's not exactly a new turn of events, commodity drives have shipped with write caching on forever. What if the drive has a battery backing?

If the drive has a batter backup, I'd argue it should report it as a write-through cache. I'm not a drive manufacturer though.

Quoted 3 lines

What if the user has an UPS?

They should enable the write-back cache if they trust the UPS. Or maybe the system should do that automatically if it's aware of the UPS.

"Policy" doesn't mean you shouldn't choose good defaults.

Quoted 6 lines

Quoted 5 lines

Quoted 3 lines

that is never the right thing to do.
There's no such thing as 'acceptable throughput',

I meant that performance is not completely destroyed. How can you even

Quoted 4 lines

How do you know it's not destroyed? Depending on your workload, it may very well be dropping your throughput by orders of magnitude.

I guess this is the crux. According to my understanding, you shouldn't see such a horrible drop, unless the application does synchronous writes explicitly, in which case it is probably worried about data safety.

Quoted 9 lines

Quoted 2 lines

compare data safety to some percent of performance?

I'm not, what I'm saying is that different people will have different opponions on what is most important. Do note that the window of corruption is really small and requires powerloss to trigger. So for most desktop users, the tradeoff is actually sane.

I agree that the window is very small, and that by eliminating software failures we get rid of the major source of data loss. What I don't know is what the performance tradeoff looks like (and I can't measure since my drives won't let me turn off the cache for some reason).

Quoted 8 lines

Quoted 7 lines

Quoted 4 lines

Additionally, write back caching is perfectly safe, if used with a barrier enabled file system in Linux.

Not all Linux filesystems are barrier enabled, AFAIK. Further, barriers don't help with O_DIRECT (right?).

Quoted 4 lines

O_DIRECT should just use FUA writes, there are safe with write back caching. I'm actually testing such a change just to gauge the performance impact.

You mean, this is not in mainline yet?

So, with this, plus barrier support for metadata and O_SYNC writes, the write-back cache should be safe?

Some googling shows that Windows XP introduced FUA for O_DIRECT and metadata writes as well.

Quoted 6 lines

Quoted 2 lines

I shouldn't need a disk array to run a database.

You are free to turn off write back caching!

What about the users who aren't on qemu-devel?

However, with your FUA change, they should be safe.

Quoted 7 lines

Quoted 3 lines

Most desktop workloads use writeback cache, so write performance is not critical.

Ehm, how do you reach that conclusion based on that statement?

Any write latency is buffered by the kernel. Write speed is main memory speed. Disk speed only bubbles up when memory is tight.

Quoted 6 lines

Quoted 3 lines

However I'd hate to see my data destroyed by a power failure, and today's large caches can hold a bunch of data.

Then you use barriers or turn write back caching off, simple as that.

I will (if I figure out how) but there may be one or two users who haven't read the scsi spec yet.

Or more correctly, I am revising my opinion of the write back cache since even when it is enabled, it is completely optional. Instead of disabling the write back cache we should use FUA and barriers, and since you are to be working on FUA, it looks like this will be resolved soon without performance/correctness compromises.

In message: <[email protected]>
Avi Kivity <[email protected]> writes:
: >> Sounds like a bug. Shouldn't Linux disable the write cache unless the : >> user explicitly enables it, if NCQ is available? NCQ should provide : >> acceptable throughput even without the write cache.
: >> : >
: > How can it be a bug? : : If it puts my data at risk, it's a bug. I can understand it for IDE,
: but not for SATA with NCQ.

So wouldn't async mounts by default be a bug too?

Quoted 2 lines

(it seems I can't turn off the write cache even without losing my data:

Use hdparm, it's an ATA drive even if Linux currently uses the scsi layer for it. Or use sysfs, there's a "cache_type" attribute in the scsi disk sysfs directory.

Quoted 2 lines

"Policy" doesn't mean you shouldn't choose good defaults.

Changing the hardware settings for this kind of behaviour IS most certainly policy.

Quoted 4 lines

I guess this is the crux. According to my understanding, you shouldn't see such a horrible drop, unless the application does synchronous writes explicitly, in which case it is probably worried about data safety.

Then you need to adjust your understanding, because you definitely will see a big drop in performance.

Quoted 7 lines

Quoted 5 lines

O_DIRECT should just use FUA writes, there are safe with write back caching. I'm actually testing such a change just to gauge the performance impact.

You mean, this is not in mainline yet?

It isn't.

Quoted 3 lines

Some googling shows that Windows XP introduced FUA for O_DIRECT and metadata writes as well.

There's a lot of other background information to understand to gauge the impact of using eg FUA for O_DIRECT in Linux as well. MS basically wrote the FUA for ATA proposal, and the original usage pattern (as far as I remember) was indeed meta data. Hence it also imposes a priority boost in most (all?) drive firmwares, since it's deemed important. So just using FUA vs non-FUA is likely to impact performance of other workloads in fairly unknown ways. FUA on non-queuing drives will also likely suck for performance, since you're basically going to be blowing a drive rev for each IO. And that hurts.

Quoted 3 lines

What about the users who aren't on qemu-devel?

It may be news to you, but it has been debated on lkml in the past as well. Not even that long ago, and I'd be surprised of lwn didn't run some article on it as well.

But I agree it's important information, but
realize that until just recently most people didn't really consider it a likely scenario in practice...

I wrote and committed the original barrier implementation in Linux in 2001, and just this year XFS made it a default mount option. After the recent debacle on this on lkml, ext4 made it the default as well.

So let me turn it around a bit - if this issue really did hit lots of people out there in real life, don't you think there would have been more noise about this and we would have made this the default years ago? So while we both agree it's a risk, it's not a huuuge risk...

Quoted 2 lines

However, with your FUA change, they should be safe.

Yes, that would make O_DIRECT safe always. Except when it falls back to buffered IO, woops...

Quoted 3 lines

Any write latency is buffered by the kernel. Write speed is main memory speed. Disk speed only bubbles up when memory is tight.

That's a nice theory, in practice that is completely wrong. You end up waiting on writes for LOTS of other reasons!

Use hdparm, it's an ATA drive even if Linux currently uses the scsi layer for it. Or use sysfs, there's a "cache_type" attribute in the scsi disk sysfs directory.

So to summarize, I think we should enable O_DSYNC by default to ensure that guest data integrity is not dependent on the host OS, and that practically speaking, cache=off is only useful for very specialized circumstances. Part of the patch I'll follow up with includes changes to the man page to document all of this for users.

Anthony Liguori writes ("[Qemu-devel] [RFC] Disk integrity in QEMU"):

Quoted 2 lines

So to summarize, I think we should enable O_DSYNC by default to ensure that guest data integrity is not dependent on the host OS, and that practically speaking, cache=off is only useful for very specialized circumstances. Part of the patch I'll follow up with includes changes to the man page to document all of this for users.

I have a patch which does this and allows the host to control the buffering with the IDE cache control facility.

I'll be resubmitting it shortly (if I manage to get round to it before going away for three weeks on Thursday lunchtime ...)

Ian.

Do you mean that the guest can control host disk cachability?

We've switched to always use O_DSYNC by default. There was a very
long thread about it including benchmarks. With the right posix-aio tuning, we can use O_DSYNC without hurting performance*.

Quoted 3 lines

We've switched to always use O_DSYNC by default. There was a very
long thread about it including benchmarks. With the right posix-aio tuning, we can use O_DSYNC without hurting performance*.

Right.

With the change in my tree, the guest can turn on the use of the host's buffer cache for writes (ie, turn off the use of O_DSYNC), using the appropriate cache control features in the IDE controller (and have write barriers with the FLUSH CACHE command).