Win2003 disk corruption with kvm-1.0. and virtio
Hi,
We currently run Openstack Essex hosts with KVM-1.0 (Ubuntu 12.04) instances with qcow2,virtio,cache=none
For Linux VMs, no trouble at all but we do observe filesystem corruption and inconsistency (missing DLLs, CHKDSK asked by EventViewer, failure at reboot) with some of our Windows 2003 SP2 64b images.
At first boot, stress tests (CrystalDiskMark 3.0.2 and intensive CHKDSK) don't show up problems. It is only appearing 6 or 12h later.
Do you have any idea on how to prevent it ? Is cache=writethrough an acceptable solution ? We don't want to leave qcow2 image format as it does allow to do live snapshots et al.
Thanks for your inputs,
-Sylvain Bauza
Hello,
Quoted 2 lines · On Tuesday 12 February 2013 15:30:37 Sylvain Bauza wrote:
We currently run Openstack Essex hosts with KVM-1.0 (Ubuntu 12.04) instances with qcow2,virtio,cache=none
The default answer is to update your qemu-kvm version: 1.0 is very old, qemu- kvm is fully merged into upstream qemu, which is currently preparing its 1.4 release.
There have been many fixes to qemi and the qcow2 handling: I know of at least one serious problem not fixed up to qemu-1.1.
Sincerely
Philipp
Signature · 4 lines
Philipp Hahn Open Source Software Engineer [email protected] Univention GmbH be open. fon: +49 421 22 232- 0 Mary-Somerville-Str.1 D-28359 Bremen fax: +49 421 22 232-99 http://www.univention.de/
Hi Philipp,
Indeed. Qemu-kvm.1.0 is pretty old but this version is the stable one for Ubuntu Precise (12.04 LTS). No backport is available for later versions, I need to install by hand.
Do you know if qemu-1.3 (with KVM support) is fully compatible with qemu-kvm.1.0 ? As I'm relying on Openstack Nova for upper hypervisor layer, it needs to be 100% matching.
Thanks,
-Sylvain
Quoted 13 lines · Philipp Hahn
Hello,
Quoted 2 lines · On Tuesday 12 February 2013 15:30:37 Sylvain Bauza wrote:
We currently run Openstack Essex hosts with KVM-1.0 (Ubuntu 12.04) instances with qcow2,virtio,cache=none
The default answer is to update your qemu-kvm version: 1.0 is very old, qemu- kvm is fully merged into upstream qemu, which is currently preparing its 1.4 release.
There have been many fixes to qemi and the qcow2 handling: I know of at least one serious problem not fixed up to qemu-1.1.Sincerely
Philipp
there are known problems, WHEN I/O "native" and cache=writethrough.
On I/O "native" put cache to "none" otherwise your data can get broken. Check Redhat Pages for that.
marko
Am 2013-02-13 10:56, schrieb Sylvain Bauza:
Quoted 34 lines
Hi Philipp,
Indeed. Qemu-kvm.1.0 is pretty old but this version is the stable one for Ubuntu Precise (12.04 LTS).
No backport is available for later versions, I need to install by hand.Do you know if qemu-1.3 (with KVM support) is fully compatible with qemu-kvm.1.0 ?
As I'm relying on Openstack Nova for upper hypervisor layer, it needs to be 100% matching.Thanks,
-SylvainQuoted 5 lines · Philipp Hahn
Hello,
Quoted 2 lines · On Tuesday 12 February 2013 15:30:37 Sylvain Bauza wrote:
We currently run Openstack Essex hosts with KVM-1.0 (Ubuntu 12.04) instances with qcow2,virtio,cache=none
Quoted 8 lines
The default answer is to update your qemu-kvm version: 1.0 is very old, qemu- kvm is fully merged into upstream qemu, which is currently preparing its 1.4 release.
There have been many fixes to qemi and the qcow2 handling: I know of at least one serious problem not fixed up to qemu-1.1.Sincerely
Philipp--
To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
[Please stop top-posting. Thank you]
Quoted 5 lines · 13.02.2013 20:03, [email protected] wrote:
there are known problems, WHEN I/O "native" and cache=writethrough.
On I/O "native" put cache to "none" otherwise your data can get broken. Check Redhat Pages for that.
Which problem is that?
And what is "I/O native" ? Maybe you mean "aio", not "I/O" ?
If the talk is about aio=native, that mode does not work for regular files, it gets "downgraded" to aio=threads automatically. So there should be nothing to change already.
Please elaborate.
/mjt
Hi,
Latest updates, I tried using :
- cache=writethrough / kvm-1.0 : errors in qcow2
- cache=none/kvm-1.3 : no errors using 'qemu-img check', but
EventViewer is complaining
I have to admit I'm lost. I cannot understand what is causing this corruption, only appearing on some Windows instances... Please find below the executable path :
117 13781 1 4 Feb13 ? 00:41:43 /usr/bin/kvm -S -M pc-1.3 -enable-kvm -m 2048 -smp 1,sockets=1,cores=1,threads=1 -name instance-0000004f -uuid 26801166-aa03-4bbc-b062-da47168a664c -nodefconfig -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/instance-0000004f.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -boot c -drive file=/var/lib/nova/instances/instance-0000004f/disk,if=none,id=drive-virtio-disk0,format=qcow2,cache=none -device virtio-blk-pci,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0 -netdev tap,fd=21,id=hostnet0,vhost=on,vhostfd=22 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=fa:16:3e:7a:a1:61,bus=pci.0,addr=0x3 -chardev file,id=charserial0,path=/var/lib/nova/instances/instance-0000004f/console.log -device isa-serial,chardev=charserial0,id=serial0 -chardev pty,id=charserial1 -device isa-serial,chardev=charserial1,id=serial1 -usb -device usb-tablet,id=input0 -vnc 192.168.1.155:2 -k fr -vga cirrus -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5
Last try, I googled and found that virtio network can be buggy. I will try to switch back to another driver and see. By the way, all these Windows instances do have virtio SCSI drivers up to date.
Quoted 13 lines · Philipp Hahn
Hello,
Quoted 2 lines · On Tuesday 12 February 2013 15:30:37 Sylvain Bauza wrote:
We currently run Openstack Essex hosts with KVM-1.0 (Ubuntu 12.04) instances with qcow2,virtio,cache=none
The default answer is to update your qemu-kvm version: 1.0 is very old, qemu- kvm is fully merged into upstream qemu, which is currently preparing its 1.4 release.
There have been many fixes to qemi and the qcow2 handling: I know of at least one serious problem not fixed up to qemu-1.1.Sincerely
Philipp
Quoted 8 lines · Sylvain Bauza
We currently run Openstack Essex hosts with KVM-1.0 (Ubuntu 12.04) instances with qcow2,virtio,cache=none
For Linux VMs, no trouble at all but we do observe filesystem corruption and inconsistency (missing DLLs, CHKDSK asked by EventViewer, failure at reboot) with some of our Windows 2003 SP2 64b images.
Quoted 6 lines
At first boot, stress tests (CrystalDiskMark 3.0.2 and intensive CHKDSK) don't show up problems. It is only appearing 6 or 12h later.
Do you have any idea on how to prevent it ? Is cache=writethrough an acceptable solution ? We don't want to leave qcow2 image format as it does allow to do live snapshots et al.
How are you taking live snapshots? qemu-img should not be used on a disk image that is currently open by a running guest, it may lead to corruption.
Stefan
Hi Stefan,
As per documentation, Nova (Openstack Compute layer) is doing a 'qemu-img convert -s' against a running instance.
http://docs.openstack.org/trunk/openstack-compute/admin/content/creating-images-from-running-instances.html
Do you think it could be our root cause ?
Btw, I tested cache=writethrough and I observed image corruption after some time ('qemu-img check' returns errors)
Thanks for your input,
-Sylvain
Quoted 20 lines · Stefan Hajnoczi
Quoted 8 lines · Sylvain Bauza
We currently run Openstack Essex hosts with KVM-1.0 (Ubuntu 12.04) instances with qcow2,virtio,cache=none
For Linux VMs, no trouble at all but we do observe filesystem corruption and inconsistency (missing DLLs, CHKDSK asked by EventViewer, failure at reboot) with some of our Windows 2003 SP2 64b images.
Quoted 6 lines
At first boot, stress tests (CrystalDiskMark 3.0.2 and intensive CHKDSK) don't show up problems. It is only appearing 6 or 12h later.
Do you have any idea on how to prevent it ? Is cache=writethrough an acceptable solution ? We don't want to leave qcow2 image format as it does allow to do live snapshots et al.
How are you taking live snapshots? qemu-img should not be used on a disk image that is currently open by a running guest, it may lead to corruption.
Stefan
Quoted 3 lines · Sylvain Bauza
As per documentation, Nova (Openstack Compute layer) is doing a 'qemu-img convert -s' against a running instance.
http://docs.openstack.org/trunk/openstack-compute/admin/content/creating-images-from-running-instances.html
That command will not corrupt the running instance because it opens the image read-only.
It is possible that the new image is corrupted since qemu-img is reading from a qcow2 file that is changing underneath it. However, the chance is small as long as the snapshot isn't deleted while qemu-img convert is running.
So this doesn't sound like the cause of the problems you are seeing.
Stefan
Interesting point you mention. Even if qcow2 is read only, the image is changing (especially, I'm running IIS with ASP support and VB DLLs) while the snapshot is taken.
As asked in a second post, I'm running with latest Windows virtio drivers, but I only apply a virtio driver update *after* running an instance, not before taking the snapshot.
What I'll try : run an instance, update the driver, stop the instance, do a qemu-img convert once the instance is stopped.
Quoted 15 lines · Stefan Hajnoczi
Quoted 3 lines · Sylvain Bauza
As per documentation, Nova (Openstack Compute layer) is doing a 'qemu-img convert -s' against a running instance.
http://docs.openstack.org/trunk/openstack-compute/admin/content/creating-images-from-running-instances.htmlThat command will not corrupt the running instance because it opens the image read-only.
It is possible that the new image is corrupted since qemu-img is reading from a qcow2 file that is changing underneath it. However, the chance is small as long as the snapshot isn't deleted while qemu-img convert is running.
So this doesn't sound like the cause of the problems you are seeing.
Stefan
Long lasting bug and huge update, but I think I got the root cause.
FYI, Windows 2003 is having a write cache enabled by default on disk drivers. Even with virtio (see driver details, policies).
As a consequence, any DLL which is open could be corrupted if we try a simple 'qemu-img convert' against the VM. The proper way to do a live snapshot is to disable the writecache (and goodbye good perfs!) and do the convert. The other way is to stop the VM, perform a 'qemu-img snapshot', then convert the snapshot.
Hope it can help other people.
-Sylvain
Quoted 25 lines · Sylvain Bauza
Interesting point you mention. Even if qcow2 is read only, the image is changing (especially, I'm running IIS with ASP support and VB DLLs) while the snapshot is taken.
As asked in a second post, I'm running with latest Windows virtio drivers, but I only apply a virtio driver update *after* running an instance, not before taking the snapshot.
What I'll try : run an instance, update the driver, stop the instance, do a qemu-img convert once the instance is stopped.
Quoted 16 lines · Stefan Hajnoczi
Quoted 4 lines · Sylvain Bauza
As per documentation, Nova (Openstack Compute layer) is doing a 'qemu-img convert -s' against a running instance.
http://docs.openstack.org/trunk/openstack-compute/admin/content/creating-images-from-running-instances.htmlThat command will not corrupt the running instance because it opens the image read-only.
It is possible that the new image is corrupted since qemu-img is reading from a qcow2 file that is changing underneath it. However, the chance is small as long as the snapshot isn't deleted while qemu-img convert is running.
So this doesn't sound like the cause of the problems you are seeing.
Stefan
Quoted 3 lines · Sylvain Bauza
Long lasting bug and huge update, but I think I got the root cause. FYI, Windows 2003 is having a write cache enabled by default on disk drivers. Even with virtio (see driver details, policies).
Hi there,
That option did you use in driver policy?
Thanks
Quoted 50 lines
As a consequence, any DLL which is open could be corrupted if we try a simple 'qemu-img convert' against the VM.
The proper way to do a live snapshot is to disable the writecache (and goodbye good perfs!) and do the convert.
The other way is to stop the VM, perform a 'qemu-img snapshot', then convert the snapshot.Hope it can help other people.
-SylvainQuoted 32 lines · Sylvain Bauza
Interesting point you mention. Even if qcow2 is read only, the image is changing (especially, I'm running IIS with ASP support and VB DLLs) while the snapshot is taken.
As asked in a second post, I'm running with latest Windows virtio drivers, but I only apply a virtio driver update *after* running an instance, not before taking the snapshot.
What I'll try : run an instance, update the driver, stop the instance, do a qemu-img convert once the instance is stopped.
Quoted 18 lines · Stefan Hajnoczi
Quoted 4 lines · Sylvain Bauza
As per documentation, Nova (Openstack Compute layer) is doing a 'qemu-img convert -s' against a running instance.
http://docs.openstack.org/trunk/openstack-compute/admin/content/creating-images-from-running-instances.htmlThat command will not corrupt the running instance because it opens the image read-only.
It is possible that the new image is corrupted since qemu-img is reading
from a qcow2 file that is changing underneath it. However, the chance is small as long as the snapshot isn't deleted while qemu-img convert is
running.So this doesn't sound like the cause of the problems you are seeing.
Stefan
Signature · 3 lines
To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 10 lines · Sylvain Bauza
We currently run Openstack Essex hosts with KVM-1.0 (Ubuntu 12.04) instances with qcow2,virtio,cache=none
For Linux VMs, no trouble at all but we do observe filesystem corruption and inconsistency (missing DLLs, CHKDSK asked by EventViewer, failure at reboot) with some of our Windows 2003 SP2 64b images.
At first boot, stress tests (CrystalDiskMark 3.0.2 and intensive CHKDSK) don't show up problems. It is only appearing 6 or 12h later.
Are you running the latest virtio-win drivers? See
http://www.linux-kvm.org/page/WindowsGuestDrivers/Download_Drivers.
Have you tested with IDE instead of virtio on the Windows guests?
Stefan