kvm + raid1 showstopper bug

I've been waiting for some response from the Ubuntu team regarding a bug on launchpad, but it appears that it isn't being taken seriously:

https://bugs.launchpad.net/ubuntu/+source/linux/+bug/745785

We are running Ubuntu Natty 11.04 with KVM with our storage on RAID10 + drbd. While we were running the resync command, we could expect an IO block during every resync. I eventually disabled resync. Yesterday we had a kernel panic, which I am presuming may trace back to KVM and its interactions with RAID10. I run KVM on some other boxes with some built-in Dell hardware RAIDs. They have been stable for well over a year on Ubuntu 10.04.

Here are the 11.04 current versions if that helps.

kvm 84+dfsg-0ubuntu16+0.14.0+noroms+0ubuntu4.4

kvm-pxe      5.4.4-7ubuntu2
qemu-kvm     0.14.0+noroms-0ubuntu4.4
drbd8-utils  8.3.9-1ubuntu1

What I'd like to know is if this bug between KVM & RAID1 is known and whether it has been addressed in newer versions of the kernel and/or KVM. I'm on the verge of buying a hardware RAID card to resolve this and I'd rather not have to deal with that. Software RAID is so much nicer to manage.

This looks interesting. Let me try to summarize, please point out if I get something wrong:

You have software RAID1 on the host, your disk images live on this device. Whenever checkarray runs on the host you find that VMs become unresponsive. Guests print warnings that a task is blocked for more than 120 seconds. Guests become unresponsive on the network.

The fact that the QEMU monitor and VNC still work mean that QEMU is not probably still running the VM. I think the guest kernel is upset, perhaps QEMU needs to do something to help these I/Os along.

Please post your qemu-kvm command-line or libvirt domain XML.

What's the easiest way to reproduce this?

Stefan

Thank you for your response Stefan.

In my case, it is drbd+RAID10, but the bug still applies. It isn't whenever checkarray runs, but whenever checkarray decides to do a resync, it will block all IO somewhere before the end of the resync. Then yes, it isn't long before the guests start to fail due to their inability to read/write.

Note that *ALL* IO is blocked, even on the host kernel. It has trouble rebooting at that point too. I have to power cycle it.

<domain type='kvm'>
<name>guestname</name> <uuid>c4cb4999-0713-dffa-32f8-1bb7278b3f5c</uuid> <memory>8388608</memory> <currentMemory>524288</currentMemory> <vcpu>1</vcpu>
<os>

    <type arch='x86_64' machine='pc-0.12'>hvm</type>
    <boot dev='hd'/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <pae/>
  </features>
  <clock offset='utc'/>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/bin/kvm</emulator>
    <disk type='block' device='disk'>
      <driver name='qemu' type='raw'/>
      <source dev='/dev/vm/guestname'/>
      <target dev='vda' bus='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04'
function='0x0'/>
    </disk>
    <interface type='bridge'>
      <mac address='52:54:00:ca:7b:70'/>
      <source bridge='br30'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03'
function='0x0'/>
    </interface>
    <serial type='pty'>
      <target port='0'/>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <input type='mouse' bus='ps2'/>
    <graphics type='vnc' port='-1' autoport='yes'/>
    <video>
      <model type='cirrus' vram='9216' heads='1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02'
function='0x0'/>
    </video>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05'

function='0x0'/>
</memballoon>
</devices>
</domain>

Based on the Launchpad bug, Ubuntu 11.04 with default packages + RAID1/RAID10 I think would have the issue eventually. I don't think it has to be a particularly IO intensive system.

I have not attempted to reproduce this yet but have taken a look at drviers/md/raid10.c resync code. md resync uses a similar mechanism for RAID1 and RAID10. While a block is being synced the entire device will force regular I/O requests to wait. There are tunables which let you rate-limit resyncing, I think this can solve your problem. Perhaps the resync is too aggressive and is impacting regular I/O so much that the guest is warning about it. See Documentation/md.txt for sync_speed_max and other sysfs attributes.

The bug report suggests qemu-kvm itself is operating fine because the guest is still executing and VNC/monitor are alive. After a while the guest warns about the stuck I/O.

Networking may become unresponsive if there is disk I/O required, e.g. ssh daemon reading keys for a user. Your best bet at testing that theory is using ICMP ping because that shouldn't involve disk I/O.

It would be interesting to start resync and then run the following on the host: time dd if=/dev/zero of=/path/to/device/tmpfile oflag=sync bs=4k count=1. You don't even need qemu-kvm for this test. I suspect this single 4 KB write to the file system will take many seconds/minutes. It would show that the problem is in the host - there is too little time for regular I/O which causes guest operating systems and applications to freak out.

Another approach to testing is running a guest without RAID resync underneath. Use dm-delay to insert an artificial delay on I/O requests (try 130 seconds). My guess is the guest operating system will react in the same way because its I/O requests take an extremely long time.

This may be dependent on hardware. I have used RAID1 to host disk images with Xen and KVM and never noticed an issue.

Stefan

Is sync_speed_max independent of dev.raid.speed_limit_max? Because I tried that to no avail.

I will try these other tests when I get a chance.

It could be a bug in the raid1/raid10 code, which is triggered by the way qemu keeps file on it (O_DIRECT or something). However there have been a *lot* of fixes to the raid code since 2.6.38 (which is the kernel version I saw referenced in the launchpad link). Please try and reproduce it with a more uptodate kernel.

I have no idea what drbd8 is and what relation it has to the kernel or kvm here.

Cheers,
Jes