Workload spikes on KVM host when doing IO on a guest...

Hi,

I am seeing high workload spikes of approx. 15 when I do IO inside a KVM guest, for instance

dd if=/dev/zero bs=1G count=1 of=hog

When I execute a similar command on the host to write a file on the same physical disk, the workload only goes to about 3.

I am using virtio on the guest with cache mode none. Also, I am using the noop IO scheduler on the guest and the deadline IO scheduler on the host. The guest is allocated a logical volume from the host.

When I execute the dd command on the guest, it finishes almost instantaneously but when I execute it on the host I have to wait for approx 10 seconds. Specifically,
on the guest I see a transfer speed of approx. 600 MB/s and on the host I get 75.9MB/s. The figure for the host is most reliable as this is close to what the hard disks can handle (WD enterprise class SATA hard disks).

What appears to be happening is that somehow it forwards all IO from the guest immediately to the host, just as if write back caching was used.

When I look at the output of 'virsh dumpxm <vmname>' I get this as part of the output which indicates that cache="none" is actually used

<driver name='qemu' type='raw' cache='none'/>
<source dev='/dev/bootdisks/sparrow'/>
<target dev='sda' bus='virtio'/>
<alias name='virtio-disk0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/> </disk>

The host is an opensuse 11.3 system (Linux falcon 2.6.34.10-0.6-default #1 SMP 2011-12-13 18:27:38 +0100 x86_64 x86_64 x86_64 GNU/Linux). The kvm version is

falcon:~ # rpm -qa | grep kvm
kvm-0.12.5-1.8.1.x86_64

Is this some known issue in this version of KVM and should I simply upgrade (or replace the host with a centos 6.2 system). Or is there a simple configuration that can fix this?

Cheers
Erik

This is not surprising. Each I/O request executes in a thread.

With logical volumes, you can use -drive ...,aio=native to avoid the threads. The load will disappear.

try dd oflag=direct to force the data to disk. No idea why the host doesn't finish instantaneously.

Write back caching is indeed used, since you did not specify oflag=direct.

Nothing is broken, so it doesn't need fixing. The high load is not an indication of anything.

error compiling committee.c: too many arguments to function

Problem is, it doesn't mean anything important. It's the count of running threads plus the count of threads uninterruptibly waiting on a mutex. It's absolutely meaningless.

No, it no longer holds a mutex. Yet it does exactly the same thing. That's an indication that the counter is meaningless.

(If the counter doesn't drop on an idle machine, that usually indicates trouble; but that's not the case)

error compiling committee.c: too many arguments to function

In the mean time I have migrated the host machine to run centos 6.2 and later 6.3. There I could use the aio=native option together with cache=none and I saw a significant drop in thw workload while doing a lot of IO on the guest.

However, after upgrading to centos 6.4, I now see huge workload spikes again, but the virtual machine configuration is the same. Also, I am seeing a high usage of swap on the host machine. It almost looks as if the aio=native option is being ignored or could it be a different problem?

Here is one of the processes:

qemu 3026 4.6 17.9 4751536 4419208 ? Sl Mar27 136:21 /usr/libexec/qemu-kvm -name sparrow -S -M rhel6.2.0 -enable-kvm -m 4096 -smp 2,sockets=2,cores=1,threads=1 -uuid 1389eb3f-8f26-686e-93c4-21267a66ec52 -nodefconfig -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/sparrow.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive file=/dev/bootdisks/sparrow,if=none,id=drive-virtio-disk0,format=raw,cache=none,aio=native -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x3,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -drive file=/dev/sparrow/disk,if=none,id=drive-virtio-disk1,format=raw,cache=none,aio=native -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x4,drive=drive-virtio-disk1,id=virtio-disk1 -netdev tap,fd=23,id=hostnet0,vhost=on,vhostfd=26 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:01:01:03,bus=pci.0,addr=0x5 -netdev tap,fd=27,id=hostnet1,vhost=on,vhostfd=28 -device virtio-net-pci,netdev=hostnet1,id=net1,mac=52:54:00:01:01:04,bus=pci.0,addr=0x7 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -vnc 127.0.0.1:2 -vga cirrus -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x6

I can also see that the swap space is exclusively used by qemu-kvm:

[root@falcon bin]# ./examineswap
...
Overall swap used: 1264924 kB
========================================
kB pid name
========================================

   575092  3026    qemu-kvm
   391468  2882    qemu-kvm
   298364  2953    qemu-kvm

(see at the end of the examineswap script).

This snapshot of the swap space is taken shortly after enabling swap and it continues to fill up the entire swap space, with still a lot of memory free

[root@falcon bin]# free
total used free shared buffers cached
Mem: 24662972 24460920 202052 0 10904048 40712
-/+ buffers/cache: 13516160 11146812
Swap: 2097144 1508516 588628

Also, I have set the swappiness to 0 but it does not help:

[root@falcon bin]# sysctl vm.swappiness
vm.swappiness = 0

Details on the OS and KVM:

[root@falcon ~]# uname -a
Linux falcon.fritz.box 2.6.32-358.2.1.el6.x86_64 #1 SMP Wed Mar 13 00:26:49 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux [root@falcon ~]# cat /etc/issue
CentOS release 6.4 (Final)
Kernel \r on an \m

[root@falcon ~]# rpm -qa | grep kvm qemu-kvm-0.12.1.2-2.355.0.1.el6.centos.2.x86_64

I am also getting a workload on the host of approximately 20 now whereas the workload on the VM doing the IO is just 6. What I am doing at this time is creating a backup over iSCSI with a target on the physical host and an initiator on the VM. The VM has created snapshot logical volumes and is copying data to the iSCSI target. More spcifically, I am using this script that I developed myself: http://wamblee.org/snapshot.html

Do you have any idea what this could be?

Cheers
Erik

PS. The examineswap script I downloaded
#!/bin/bash

    # find-out-what-is-using-your-swap.sh
    # -- Get current swap usage for all running processes
    # --
    # -- rev.0.3, 2012-09-03, Jan Smid          - alignment and 
intendation, sorting
    # -- rev.0.2, 2012-08-09, Mikko Rantalainen - pipe the output to 
"sort -nk3" to get sorted output
    # -- rev.0.1, 2011-05-27, Erik Ljungstrom   - initial version

SCRIPT_NAME=basename $0;
SORT="kb"; # {pid|kB|name} as first parameter, [default: kb]
[ "$1" != "" ] && { SORT="$1"; }

[ ! -x which mktemp ] && { echo "ERROR: mktemp is not available!"; exit; } MKTEMP=which mktemp;
TMP=${MKTEMP} -d;
[ ! -d "${TMP}" ] && { echo "ERROR: unable to create temp dir!"; exit; }

SUM=0;

OVERALL=0;
    echo "${OVERALL}" > ${TMP}/${SCRIPT_NAME}.overal;

for DIR in find /proc/ -maxdepth 1 -type d -regex "^/proc/[0-9]+"; do

    PID=`echo $DIR | cut -d / -f 3`
    PROGNAME=`ps -p $PID -o comm --no-headers`

    for SWAP in `grep Swap $DIR/smaps 2>/dev/null| awk '{ print $2 }'`
    do
        let SUM=$SUM+$SWAP
    done

    if (( $SUM > 0 ));
    then
        echo -n ".";
        echo -e "${PID}\t${SUM}\t${PROGNAME}" >> ${TMP}/${SCRIPT_NAME}.pid;
        echo -e "${SUM}\t${PID}\t${PROGNAME}" >> ${TMP}/${SCRIPT_NAME}.kb;
        echo -e "${PROGNAME}\t${SUM}\t${PID}" >> 
${TMP}/${SCRIPT_NAME}.name;
    fi
    let OVERALL=$OVERALL+$SUM
    SUM=0

done
echo "${OVERALL}" > ${TMP}/${SCRIPT_NAME}.overal;
echo;
echo "Overall swap used: ${OVERALL} kB";
echo "========================================";
case "${SORT}" in

    name )
        echo -e "name\tkB\tpid";
        echo "========================================";
        cat ${TMP}/${SCRIPT_NAME}.name|sort -r;
        ;;

    kb )
        echo -e "kB\tpid\tname";
        echo "========================================";
        cat ${TMP}/${SCRIPT_NAME}.kb|sort -rh;
        ;;

    pid | * )
        echo -e "pid\tkB\tname";
        echo "========================================";
        cat ${TMP}/${SCRIPT_NAME}.pid|sort -rh;
        ;;

esac
rm -fR "${TMP}/";