Qemu disk images on BTRFS suffer checksum errors
Hello,
I'm about to migrate from VirtualBox to Qemu+VGA-Passthrough. All my virtual disk images are stored in a BTRFS subvolume on-top of a MDRAID 1. The host runs kernel 3.10, and Qemu 1.5.1. The Testing-VM is a Windows 7 64bit, using a RAW virtio disk with cache=none, same happens for qcow2, though.
Using VirtualBox and in the past Vmware workstation I never had issues with corrupted diskimages, but now with Qemu all tries ended up with lots of errors like:
[ 4871.863009] BTRFS info (device md10): csum failed ino 687 off 46213922816 csum 3817758510 private 402306600 [ 4872.481013] BTRFS info (device md10): csum failed ino 687 off 46213922816 csum 3817758510 private 402306600 [ 4904.055514] BTRFS info (device md10): csum failed ino 687 off 46213922816 csum 4060166193 private 402306600 [ 4904.748130] BTRFS info (device md10): csum failed ino 687 off 46213922816 csum 4060166193 private 402306600 [ 4904.987540] BTRFS info (device md10): csum failed ino 687 off 46213922816 csum 3817758510 private 402306600 [ 4905.024700] BTRFS info (device md10): csum failed ino 687 off 46213922816 csum 3817758510 private 402306600 [ 4932.497793] BTRFS info (device md10): csum failed ino 687 off 46213922816 csum 4060166193 private 402306600 [ 4932.533634] BTRFS info (device md10): csum failed ino 687 off 46213922816 csum 4060166193 private 402306600
Trying to copy the disk image elsewhere causes I/O errors at some point.
I found a thread about the issue (http://comments.gmane.org/gmane.comp.file-systems.btrfs/20538) and also a bug report against Qemu from Josef Bacik describing the exact same problem: https://bugzilla.redhat.com/show_bug.cgi?id=693530 - Josef states it should be fixed since quite a while.
Is this a regression in BTRFS, a problem with my setup (md raid1 layer below btrfs), or (still) a bug in Qemu? Would cache=writethrough or writeback be an option with BTRFS?
Thanks in advance for any input.
Best regards,
Thomas
Quoted 37 lines · Thomas Kuther
Hello,
I'm about to migrate from VirtualBox to Qemu+VGA-Passthrough. All my virtual disk images are stored in a BTRFS subvolume on-top of a MDRAID 1. The host runs kernel 3.10, and Qemu 1.5.1. The Testing-VM is a Windows 7 64bit, using a RAW virtio disk with cache=none, same happens for qcow2, though.
Using VirtualBox and in the past Vmware workstation I never had issues with corrupted diskimages, but now with Qemu all tries ended up with lots of errors like:
[ 4871.863009] BTRFS info (device md10): csum failed ino 687 off 46213922816 csum 3817758510 private 402306600 [ 4872.481013] BTRFS info (device md10): csum failed ino 687 off 46213922816 csum 3817758510 private 402306600 [ 4904.055514] BTRFS info (device md10): csum failed ino 687 off 46213922816 csum 4060166193 private 402306600 [ 4904.748130] BTRFS info (device md10): csum failed ino 687 off 46213922816 csum 4060166193 private 402306600 [ 4904.987540] BTRFS info (device md10): csum failed ino 687 off 46213922816 csum 3817758510 private 402306600 [ 4905.024700] BTRFS info (device md10): csum failed ino 687 off 46213922816 csum 3817758510 private 402306600 [ 4932.497793] BTRFS info (device md10): csum failed ino 687 off 46213922816 csum 4060166193 private 402306600 [ 4932.533634] BTRFS info (device md10): csum failed ino 687 off 46213922816csum 4060166193 private 402306600
Trying to copy the disk image elsewhere causes I/O errors at some point.
I found a thread about the issue
(http://comments.gmane.org/gmane.comp.file-systems.btrfs/20538) and also a bug report against Qemu from Josef Bacik describing the exact same problem:
https://bugzilla.redhat.com/show_bug.cgi?id=693530 - Josef states it should be fixed since quite a while.
Quoted 4 lines
Is this a regression in BTRFS, a problem with my setup (md raid1 layer below btrfs), or (still) a bug in Qemu?
Would cache=writethrough or writeback be an option with BTRFS?
So there were two aspects to that bug, one is the thing I describe where we get the same buffer for two parts of an iovec on reads. That part has been fixed. The second part is where the application will modify the page while it's in flight, and that hasn't been fixed. We have a few options here
1) Always double buffer direct io. Kind of defeats the purpose of direct io.
2) Check the buffer after we've written it to see if it matches the csum we put down, if not double buffer it and send it down again. This makes you checksum the page twice and punishes O_DIRECT users that behave.
I opted for #3 and let this sort of thing happen. So you can get around it by doing nodatacow for that particular image which will disable checksumming for just that file, or you can use cache=writethrough/writeback and that will use buffered io. FYI this doesn't happen on _all_ qemu, just on guest OS'es that don't provide stable pages, so Windows or like old RHEL versions that are on ext3. Thanks,
Josef
Am 08.07.2013 15:20, schrieb Josef Bacik:
Quoted 54 lines · Thomas Kuther
Quoted 37 lines
Hello,
I'm about to migrate from VirtualBox to Qemu+VGA-Passthrough. All my virtual disk images are stored in a BTRFS subvolume on-top of a MDRAID 1. The host runs kernel 3.10, and Qemu 1.5.1. The Testing-VM is a Windows 7 64bit, using a RAW virtio disk with cache=none, same happens for qcow2, though.
Using VirtualBox and in the past Vmware workstation I never had issues with corrupted diskimages, but now with Qemu all tries ended up with lots of errors like:
[ 4871.863009] BTRFS info (device md10): csum failed ino 687 off 46213922816 csum 3817758510 private 402306600 [ 4872.481013] BTRFS info (device md10): csum failed ino 687 off 46213922816 csum 3817758510 private 402306600 [ 4904.055514] BTRFS info (device md10): csum failed ino 687 off 46213922816 csum 4060166193 private 402306600 [ 4904.748130] BTRFS info (device md10): csum failed ino 687 off 46213922816 csum 4060166193 private 402306600 [ 4904.987540] BTRFS info (device md10): csum failed ino 687 off 46213922816 csum 3817758510 private 402306600 [ 4905.024700] BTRFS info (device md10): csum failed ino 687 off 46213922816 csum 3817758510 private 402306600 [ 4932.497793] BTRFS info (device md10): csum failed ino 687 off 46213922816 csum 4060166193 private 402306600 [ 4932.533634] BTRFS info (device md10): csum failed ino 687 off 46213922816csum 4060166193 private 402306600
Trying to copy the disk image elsewhere causes I/O errors at some point.
I found a thread about the issue
(http://comments.gmane.org/gmane.comp.file-systems.btrfs/20538) and also a bug report against Qemu from Josef Bacik describing the exact same problem:
https://bugzilla.redhat.com/show_bug.cgi?id=693530 - Josef states it should be fixed since quite a while.Quoted 4 lines
Is this a regression in BTRFS, a problem with my setup (md raid1 layer below btrfs), or (still) a bug in Qemu?
Would cache=writethrough or writeback be an option with BTRFS?So there were two aspects to that bug, one is the thing I describe where we get the same buffer for two parts of an iovec on reads. That part has been fixed. The second part is where the application will modify the page while it's in flight, and that hasn't been fixed. We have a few options here
1) Always double buffer direct io. Kind of defeats the purpose of direct io.
2) Check the buffer after we've written it to see if it matches the csum we put down, if not double buffer it and send it down again. This makes you checksum the page twice and punishes
O_DIRECTusers that behave.
Quoted 9 lines
I opted for #3 and let this sort of thing happen. So you can get around it by doing nodatacow for that particular image which will disable checksumming for just that file, or you can use cache=writethrough/writeback and that will use buffered io. FYI this doesn't happen on _all_ qemu, just on guest OS'es that don't provide stable pages, so Windows or like old RHEL versions that are on ext3. Thanks,
Josef
Thanks very much for the explanation, Josef.
I opted for 3), too. Used chattr +C on the directory that is meant for holding the qemu image(s), and re-created the RAW image in there (so it has nodatacow flag set now)
So far, no issues. Perfect.
Thanks again.
~Tom