kvm + ceph performance issues

hi,

I'm running a virtual machine with the following command:

LC_ALL=C
PATH=/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/sbin:/sbin:/bin
QEMU_AUDIO_DRV=none /usr/bin/kvm -S -M pc-1.0 -enable-kvm -m 4096 -smp
2,sockets=2,cores=1,threads=1 -name korfu_ceph -uuid
a9131b8f-d087-26f4-2ca9-018505f11838 -nodefconfig -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/korfu_ceph.monitor,server,nowait
-mon chardev=charmonitor,id=monitor,mode=control -rtc base=localtime
-no-shutdown -device lsi,id=scsi0,bus=pci.0,addr=0x4 -drive file=rbd:rd/korfu:rbd_cache=1:mon_host=rd-clusternode21\:6789\;rd-clusternode22\:6789,if=none,id=drive-ide0-0-0,format=raw -device
ide-drive,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0,bootindex=1
-usb -vnc 127.0.0.1:0 -vga std -device
virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5

kvm-version is that from ubuntu LTS: 1.0+noroms-0ubuntu14.7

When I read or write big files the system basically gets unusable (mouse-cursor in VNC jerks across the screen, i/o is mostly on blocking). I know it is related to ceph in a way, but also to KVM, as it seems that there a lot of IRQ's happening (or how do you explain the mourse cursor in VNC jerking and lagging behind time?_) . Ceph Mailing List doesn't really help. High CPU load doesn't hurt the machine, it's only Harddisk I/O. Oh, and the main host running kvm doesn't really suffer, too. some i/o waiting, but not really swapping or something.

Here's my libvirt-config if it is of any help:
http://pastie.org/6411055

any hints would REALLY be appreciated...

Wolfgang

Please try using virtio-blk instead of IDE.

If the guest still jerks try using the Linux rbd block driver instead of QEMU -drive rbd:. I haven't used Ceph much but there should be documentation on attaching a RADOS block device to your Linux host. Tell QEMU to use the RADOS block device like a regular file (you are now using the kernel driver instead of QEMU code to talk to the Ceph cluster).

Please let us know the outcome. If you find that virtio-blk does not make much difference but using the kernel rbd driver does, then this suggests there is a bug in QEMU's block/rbd.c.

Stefan

there is no difference if I use virtio-blk.

This doesn't work yet on the linux-kernel with format 2 images. Format 2 is capable of read/write snapshots and cooler cloning-stuff. I need this for backup-reasons.

I'm pretty sure there is, other ceph users reported the same problems that I've found. The ceph developers are looking into it.

Wolfgang