Poor performance with qemu
Hi, I'm using KVM, and the virtual disk (a 20 GB file using the "raw" qemu format according to virt-manager and, of course, placed on a btrfs filesystem, running the latest mainline git) is awfully slow, no matter what OS is running inside the VM. The PCBSD installer says it's copying data at a 40-50 KB/s rate. Is someone using KVM and having better numbers than me? How can I help to debug this workload?
Quoted 6 lines · Diego Calleja
Hi, I'm using KVM, and the virtual disk (a 20 GB file using the "raw" qemu format according to virt-manager and, of course, placed on a btrfs filesystem, running the latest mainline git) is awfully slow, no matter what OS is running inside the VM. The PCBSD installer says it's copying data at a 40-50 KB/s rate. Is someone using KVM and having better numbers than me? How can I help to debug this workload?
The problem is that qemu uses O_SYNC by default, which makes btrfs do log commits for every write.
Once the O_DIRECT read patch is in, you can switch to that, or tell qemu to use a writeback cache instead.
-chris
Quoted 9 lines · Chris Mason
Quoted 7 lines · Diego Calleja
Hi, I'm using KVM, and the virtual disk (a 20 GB file using the "raw" qemu format according to virt-manager and, of course, placed on a btrfs filesystem, running the latest mainline git) is awfully slow, no matter what OS is running inside the VM. The PCBSD installer says it's copying data at a 40-50 KB/s rate. Is someone using KVM and having better numbers than me? How can I help to debug this workload?
Quoted 3 lines
The problem is that qemu uses
O_SYNCby default, which makes btrfs do log commits for every write.
Problem is, btrfs takes the 50 KB/s guest rate and inflates it to something much larger (megabytes/sec). Are there plans to reduce the amount of O_SYNC overhead writes?
I saw this too, but with 2.6.31 or 2.6.32 IIRC.
Quoted 3 lines
Once the
O_DIRECTread patch is in, you can switch to that, or tell qemu to use a writeback cache instead.
Even with writeback qemu will issue a lot of fsyncs.
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 16 lines · Avi Kivity
Quoted 9 lines · Chris Mason
Quoted 7 lines · Diego Calleja
Hi, I'm using KVM, and the virtual disk (a 20 GB file using the "raw" qemu format according to virt-manager and, of course, placed on a btrfs filesystem, running the latest mainline git) is awfully slow, no matter what OS is running inside the VM. The PCBSD installer says it's copying data at a 40-50 KB/s rate. Is someone using KVM and having better numbers than me? How can I help to debug this workload?
Quoted 3 lines
The problem is that qemu uses
O_SYNCby default, which makes btrfs do log commits for every write.Problem is, btrfs takes the 50 KB/s guest rate and inflates it to something much larger (megabytes/sec). Are there plans to reduce the amount of
O_SYNCoverhead writes?
Quoted 8 lines
I saw this too, but with 2.6.31 or 2.6.32 IIRC.
Quoted 3 lines
Once the
O_DIRECTread patch is in, you can switch to that, or tell qemu to use a writeback cache instead.Even with writeback qemu will issue a lot of fsyncs.
My understanding was that with cache=writeback qemu shouldn't issue any fsyncs at all, but I could be wrong.
Gordan
Quoted 8 lines · Avi Kivity
Quoted 7 lines · Chris Mason
Quoted 6 lines · Diego Calleja
Hi, I'm using KVM, and the virtual disk (a 20 GB file using the "raw" qemu format according to virt-manager and, of course, placed on a btrfs filesystem, running the latest mainline git) is awfully slow, no matter what OS is running inside the VM. The PCBSD installer says it's copying data at a 40-50 KB/s rate. Is someone using KVM and having better numbers than me? How can I help to debug this workload?
Quoted 8 lines
Quoted 2 lines
The problem is that qemu uses
O_SYNCby default, which makes btrfs do log commits for every write.Problem is, btrfs takes the 50 KB/s guest rate and inflates it to something much larger (megabytes/sec). Are there plans to reduce the amount of
O_SYNCoverhead writes?I saw this too, but with 2.6.31 or 2.6.32 IIRC.
With O_DIRECT the writeback rates are very reasonable. I'll work up a way to pass the barrier down from the guest to btrfs to force logging of updated metadata when required.
Quoted 5 lines
Quoted 3 lines
Once the
O_DIRECTread patch is in, you can switch to that, or tell qemu to use a writeback cache instead.Even with writeback qemu will issue a lot of fsyncs.
Oh, I didn't see that when I was testing, when does it fsync?
-chris
Quoted 8 lines · Chris Mason
Quoted 6 lines
Quoted 4 lines
Once the
O_DIRECTread patch is in, you can switch to that, or tell qemu to use a writeback cache instead.Even with writeback qemu will issue a lot of fsyncs.
Oh, I didn't see that when I was testing, when does it fsync?
When it updates qcow2 metadata or when the guest issues a barrier. It's relatively new. I have a patch that introduces cache=volatile somewhere.
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 10 lines · Avi Kivity
Quoted 5 lines · Chris Mason
Quoted 4 lines
Quoted 3 lines
Once the
O_DIRECTread patch is in, you can switch to that, or tell qemu to use a writeback cache instead.Even with writeback qemu will issue a lot of fsyncs.
Oh, I didn't see that when I was testing, when does it fsync?
When it updates qcow2 metadata or when the guest issues a barrier. It's relatively new. I have a patch that introduces cache=volatile somewhere.
Ok, that's actually perfect. I'll retest with that if I can get qemu-git to work here.
-chris
Quoted 2 lines · Avi Kivity
When it updates qcow2 metadata or when the guest issues a barrier. It's relatively new. I have a patch that introduces cache=volatile somewhere.
qcow2 does not issues any fsyncs by itself, it only passes throught the guests ones. The only other placess issueing fsyncs is commit a COW image back to the base image, and on migreation.
Quoted 9 lines · Christoph Hellwig
Quoted 3 lines · Avi Kivity
When it updates qcow2 metadata or when the guest issues a barrier. It's relatively new. I have a patch that introduces cache=volatile somewhere.
qcow2 does not issues any fsyncs by itself, it only passes throught the guests ones. The only other placess issueing fsyncs is commit a COW image back to the base image, and on migreation.
Shouldn't it do that then? What's the point of fsyncing guest data if qcow2 metadata is volatile?
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 2 lines · Avi Kivity
Shouldn't it do that then? What's the point of fsyncing guest data if qcow2 metadata is volatile?
Not my territory - but in the end getting qcow2 as-is solid in face of crashes will be an uphill battel - I'd rather recommend not using it.
Quoted 3 lines · Chris Mason
With
O_DIRECTthe writeback rates are very reasonable. I'll work up a way to pass the barrier down from the guest to btrfs to force logging of updated metadata when required.
Barriers are implemented in the guest kernel using queue drains and cache flush commands. Qemu maps the cache flush to fdatasync.
Quoted 6 lines
Quoted 4 lines
Quoted 2 lines
Once the
O_DIRECTread patch is in, you can switch to that, or tell qemu to use a writeback cache instead.Even with writeback qemu will issue a lot of fsyncs.
Oh, I didn't see that when I was testing, when does it fsync?
When the guest issues a barrier (and it's actually a fdatasync)