Very high memory usage with KVM

Hi all!

I have an installation with Ubuntu Hardy Heron server amd64 with KVM-62 from Ubuntu repositories installed on an HP Proliant DL380 G5 with two Xeon E5405 quadcore processors and 16 GiB of RAM which has six VMs with the following configuration of memory:

Hostname | RAM
===============+===============

Ganimedes      |    2 GiB
Os             |    1 GiB
Aprender       |    2 GiB
Aps0           |    2 GiB
Aps2           |    4 GiB
Ratatoskr      |    4 GiB

===============+===============
TOTAL | 15 GiB

Initially the host was created with a swap partition of 1 GiB (more 1 GiB than was free for use of host) but this amount with the time remained short and I had to add a LV of 7 GiB to be used with swap, being now a total of 8 GiB of swap of which at this moment I have only a 9% free. Is 'normal' this use of memory?

root@ss02:~# ps -e --sort -rss -Ho user,start_time,pid,pcpu,pmem,rss,size,vsz,args
USER     START   PID %CPU %MEM   RSS    SZ    VSZ COMMAND
[...]
root     Jul06 27471 52.3 24.4 4023232 4292200 4350296   kvm <ratatoskr>
root     Jul24  9955  137 23.8 3923620 4308592 4350308   kvm <aps2>
root     Jul06  8751  5.8  8.3 1368228 2171808 2229888   kvm <aps0>
root     Jul07  8565  2.7  5.2 862844 2204704 2246416   kvm <aprender>
root     Apr22  7842  0.6  3.6 600072 2172056 2230136   kvm <ganimedes>
root     Jul01  7944  0.6  2.0 334860 1119916 1177996   kvm <os>

root@ss02:~# free
             total       used       free     shared    buffers     cached
Mem:      16463388   16377844      85544          0     894216      66328
-/+ buffers/cache:   15417300    1046088
Swap:      8319948    7621916     698032

Updating to KVM-84 or superior can improve this situation?

Thanks in advance.

Regards,
Daniel

Fingerprint: BFB3 08D6 B4D1 31B2 72B9  29CE 6696 BF1B 14E6 1D37
Powered by Debian GNU/Linux Squeeze - Linux user #188.598

What is the storage configuration? Are you using qcow2? What are the image logical and physical sizes?

What is the host kernel (uname -a)?

error compiling committee.c: too many arguments to function

Hi Avi.

The host machine has 8 x 300 GiB SAS disk in RAID 5 by hardware (7 disks with 1 spare) on 1 logicaldrive. The partitioning scheme of host is the following one:

root@ss02:~# fdisk -l /dev/cciss/c0d0

Disk /dev/cciss/c0d0: 1799.7 GB, 1799797127168 bytes 255 heads, 63 sectors/track, 218812 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Disk identifier: 0x000af3c3

           Device Boot      Start         End      Blocks   Id  System
/dev/cciss/c0d0p1               1         122      979933+  82  Linux swap / Solaris
/dev/cciss/c0d0p2   *         123        1338     9767520   83  Linux
/dev/cciss/c0d0p3            1339      218812  1746859905   8e  Linux LVM

I'm not using qcow2 files. The /dev/cciss/c0d0p3 partition is a physical volume that maintains the logical volumes that are used for VM's disks:

root@ss02:~# pvs
  PV                VG   Fmt  Attr PSize PFree
  /dev/cciss/c0d0p3 vm   lvm2 a-   1,63T 1,13T

The disks for the VMs have these sizes:

  aprender-raiz   vm   -wi-ao   8,00G
  aprender-space  vm   -wi-ao  20,00G

  aps0-raiz       vm   -wi-ao   7,00G
  aps0-space      vm   -wi-ao  10,00G

  aps2-cache      vm   -wi-ao  20,00G
  aps2-index      vm   -wi-ao  10,00G
  aps2-raiz       vm   -wi-ao   7,00G
  aps2-space      vm   -wi-ao  10,00G

ganimedes-raiz vm -wi-ao 5,00G
ganimedes-space vm -wi-ao 10,00G

  os-disk         vm   -wi-ao   6,00G
  os-mailbox      vm   -wi-ao 150,00G
  os-space        vm   -wi-ao  10,00G

ratatoskr-raiz vm -wi-ao 8,00G
ratatoskr-space vm -wi-ao 200,00G

With respect to the internal partitioning scheme for each one of the VMs, it is the following one:

* Aprender:

aprender:~# fdisk -l /dev/hda

Disk /dev/hda: 8589 MB, 8589934592 bytes 255 heads, 63 sectors/track, 1044 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot      Start         End      Blocks   Id  System
/dev/hda1               1         486     3903763+  82  Linux swap / Solaris
/dev/hda2             487        1044     4482135   83  Linux
aprender:~#
aprender:~#
aprender:~# fdisk -l /dev/hdb

Disk /dev/hdb: 21.4 GB, 21474836480 bytes 255 heads, 63 sectors/track, 2610 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot      Start         End      Blocks   Id  System
/dev/hdb1               1        2610    20964793+  83  Linux

* Aps0:

[root@aps:~]
$fdisk -l /dev/hda

Disco /dev/hda: 7516 MB, 7516192768 bytes 255 heads, 63 sectors/track, 913 cylinders Units = cilindros of 16065 * 512 = 8225280 bytes Disk identifier: 0x00039c2a

Disposit. Inicio    Comienzo      Fin      Bloques  Id  Sistema
/dev/hda1               1         365     2931831   82  Linux swap / Solaris
/dev/hda2             366         913     4401810   83  Linux

[root@aps:~]
$
[root@aps:~]
$fdisk -l /dev/hdb

Disco /dev/hdb: 10.7 GB, 10737418240 bytes 255 heads, 63 sectors/track, 1305 cylinders Units = cilindros of 16065 * 512 = 8225280 bytes Disk identifier: 0x00087cc1

Disposit. Inicio    Comienzo      Fin      Bloques  Id  Sistema
/dev/hdb1               1        1305    10482381   83  Linux

* Aps2

aps2:~# fdisk -l /dev/hda

Disco /dev/hda: 7516 MB, 7516192768 bytes 255 heads, 63 sectors/track, 913 cylinders Units = cilindros of 16065 * 512 = 8225280 bytes Disk identifier: 0x0004b793

Disposit. Inicio    Comienzo      Fin      Bloques  Id  Sistema
/dev/hda1               1         365     2931831   82  Linux swap /

Solaris
/dev/hda2 366 913 4401810 83 Linux
aps2:~#
aps2:~#

aps2:~# fdisk -l /dev/hdb

Disco /dev/hdb: 10.7 GB, 10737418240 bytes 255 heads, 63 sectors/track, 1305 cylinders Units = cilindros of 16065 * 512 = 8225280 bytes Disk identifier: 0x0009491d

Disposit. Inicio    Comienzo      Fin      Bloques  Id  Sistema
/dev/hdb1               1        1305    10482381   83  Linux
aps2:~#
aps2:~#
aps2:~# fdisk -l /dev/hdc

Disco /dev/hdc: 10.7 GB, 10737418240 bytes 255 heads, 63 sectors/track, 1305 cylinders Units = cilindros of 16065 * 512 = 8225280 bytes Disk identifier: 0x00000000

Disposit. Inicio    Comienzo      Fin      Bloques  Id  Sistema
/dev/hdc1               1        1305    10482381   83  Linux
aps2:~#
aps2:~#
aps2:~# fdisk -l /dev/hdd

Disco /dev/hdd: 21.4 GB, 21474836480 bytes 255 heads, 63 sectors/track, 2610 cylinders Units = cilindros of 16065 * 512 = 8225280 bytes Disk identifier: 0x00000000

Disposit. Inicio    Comienzo      Fin      Bloques  Id  Sistema
/dev/hdd1               1        2610    20964793+  83  Linux

* Ganimedes:

[root@ganimedes:~]
$fdisk -l /dev/hda

Disco /dev/hda: 5368 MB, 5368709120 bytes 255 heads, 63 sectors/track, 652 cylinders Units = cilindros of 16065 * 512 = 8225280 bytes Disk identifier: 0x0002cad1

Disposit. Inicio    Comienzo      Fin      Bloques  Id  Sistema
/dev/hda1               1         122      979933+  82  Linux swap /

Solaris
/dev/hda2 123 652 4257225 83 Linux
[root@ganimedes:~]
$
[root@ganimedes:~]
$fdisk -l /dev/hdb

Disco /dev/hdb: 10.7 GB, 10737418240 bytes 255 heads, 63 sectors/track, 1305 cylinders Units = cilindros of 16065 * 512 = 8225280 bytes Disk identifier: 0x0007e07e

Disposit. Inicio    Comienzo      Fin      Bloques  Id  Sistema
/dev/hdb1               1        1305    10482381   83  Linux

* Os:

os:~# fdisk -l /dev/hda

Disco /dev/hda: 6442 MB, 6442450944 bytes 255 heads, 63 sectors/track, 783 cylinders Units = cilindros of 16065 * 512 = 8225280 bytes Disk identifier: 0x000a78a9

Disposit. Inicio    Comienzo      Fin      Bloques  Id  Sistema
/dev/hda1               1         243     1951866   82  Linux swap /

Solaris
/dev/hda2 244 783 4337550 83 Linux
os:~#
os:~#

os:~# fdisk -l /dev/hdb

Disco /dev/hdb: 10.7 GB, 10737418240 bytes 255 heads, 63 sectors/track, 1305 cylinders Units = cilindros of 16065 * 512 = 8225280 bytes Disk identifier: 0x000dffb2

Disposit. Inicio    Comienzo      Fin      Bloques  Id  Sistema
/dev/hdb1               1        1305    10482381   83  Linux
os:~#
os:~#
os:~# fdisk -l /dev/hdc

Disco /dev/hdc: 161.0 GB, 161061273600 bytes 255 heads, 63 sectors/track, 19581 cylinders Units = cilindros of 16065 * 512 = 8225280 bytes Disk identifier: 0x90c7dc67

Disposit. Inicio    Comienzo      Fin      Bloques  Id  Sistema
/dev/hdc1               1       19581   157284351   83  Linux

* Ratatosk:

ratatoskr:~# fdisk -l /dev/hda

Disk /dev/hda: 8589 MB, 8589934592 bytes 255 heads, 63 sectors/track, 1044 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot      Start         End      Blocks   Id  System
/dev/hda1               1         486     3903763+  82  Linux swap /

Solaris
/dev/hda2 487 1044 4482135 83 Linux
ratatoskr:~#
ratatoskr:~#

ratatoskr:~# fdisk -l /dev/hdb

Disk /dev/hdb: 214.7 GB, 214748364800 bytes 255 heads, 63 sectors/track, 26108 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot      Start         End      Blocks   Id  System
/dev/hdb1               1       26108   209712478+  83  Linux

root@ss02:~# uname -a
Linux ss02 2.6.24-19-server #1 SMP Wed Aug 20 18:43:06 UTC 2008 x86_64 GNU/Linux

Thanks for your reply.

Regards,
Daniel

Fingerprint: BFB3 08D6 B4D1 31B2 72B9  29CE 6696 BF1B 14E6 1D37
Powered by Debian GNU/Linux Squeeze - Linux user #188.598

In this case there should be no excessive memory usage. qcow2 could use extra memory, especially on older qemu-kvm versions (or images created with older qemu-img versions).

kvm memory management with pre 2.6.27 host kernels is pretty weak. Using a newer host kernel (and newer kvm) may solve this problem.

error compiling committee.c: too many arguments to function

Avi

Initially I am going to see how it improves the situation upgrading to KVM-84 of backports of Hardy Heron, since at the moment last kernel available for Hardy is the one I has commented.

I would like to know if the newest versions of KVM published in the official site of the project solve a bug recently reported in Ubuntu Launchpad [1]. Also I was observing errors of type 'to swapper Not tainted' or 'java Not tainted' in the VM (aps2, with a high rate of I/O) and that I've commented in a previous message sent to the list [2]. I would want to know if you could indicate to me if this can be due to KVM bug that would be solved in a later version.

Thanks for so quick reply.

Regards,
Daniel

[1] https://bugs.launchpad.net/ubuntu/+source/kvm/+bug/359447
[2] http://thread.gmane.org/gmane.comp.emulators.kvm.devel/37631

Fingerprint: BFB3 08D6 B4D1 31B2 72B9  29CE 6696 BF1B 14E6 1D37
Powered by Debian GNU/Linux Squeeze - Linux user #188.598

My guess is that it is due to poor swapping with pre-2.6.27 hosts. 15 GB used out of 16GB total is just 6% reserve, which may be a bit too low. With a 2.6.27 host some small amount of memory would be swapped out, before that you'd see thrashing. Another way to check is to drop one guest (or reduce total memory needed by 1GB) and see if you get the same results or if things improve.

error compiling committee.c: too many arguments to function

Hi Avi.

I was testing in another equipment of similar characteristics with 4 VMs of 3.5 GiB of RAM on LVM and rather less the 16 GB of RAM in use in the host machine and 4 GIB available of the 8 GIB of swap, and after to update to KVM-88 the use of virtual memory improved enough.

At the moment these are the statistics of memory usage:

root@ss03:~# free
             total       used       free     shared    buffers     cached
Mem:      16463396    6700900    9762496          0    2805756     391884
-/+ buffers/cache:    3503260   12960136
Swap:      8319948          0    8319948

Still I didn't update the kernel, so I am using KVM-88 with kernel 2.6.24-19. I would like to know if this can have some disadvantage, since in the official site of the project I was reading that kernel has to be 2.6.25 or newer to run the kvm 76 userspace (or any newer release).

The reason by which I didn't update kernel is because I was with some problems with the network interface on the host machine. Perhaps this problem is off-topic, but if somebody can help me with this, then it would contribute whereupon it can improve the KVM performance.

I've downloaded to the source code of 2.6.30.3 and I compiled it with:

# fakeroot make-kpkg clean
# fakeroot make-kpkg --initrd --append-to-version=-dgb kernel_image kernel_headers

Although the compilation finishes without problems, I am having a problem booting the operating system. The interfaces fail to load. This equipment is a Proliant DL380 G5 with two NetXtreme II BCM5708 Ethernet Gigabit interfaces.

With kernel of the Ubuntu Hardy repositories I did not have any kind of problem, but with kernel compiled by myself I obtain the following error message during the boot that it suggests cannot accede to firmware of this network interface:

Aug  4 17:38:39 ss03 kernel: [    4.750023] bnx2 0000:03:00.0: firmware: requesting bnx2/bnx2-mips-06-4.6.16.fw
Aug  4 17:38:39 ss03 kernel: [    4.751070] bnx2: Can't load firmware file "bnx2/bnx2-mips-06-4.6.16.fw"
Aug  4 17:38:39 ss03 kernel: [    4.751141] bnx2 0000:03:00.0: PCI INT A disabled
Aug  4 17:38:39 ss03 kernel: [    4.751167] bnx2: probe of 0000:03:00.0 failed with error -2
Aug  4 17:38:39 ss03 kernel: [    4.751244] bnx2 0000:05:00.0: PCI INT A -> GSI 17 (level, low) -> IRQ 17

The peculiar thing is that if I do:

# rmmod bnx2
# modprobe bnx2

Yes it loads the interfaces. I read something of this in a bug report of Debian GNU/Linux [1] and tried with the suggestion to do:

# update-initramfs -u

The files seem to be:

# find / | grep bnx
/boot/lib/modules/2.6.30.3-dgb/kernel/drivers/net/bnx2.ko
/sys/bus/pci/drivers/bnx2
/sys/bus/pci/drivers/bnx2/0000:03:00.0
/sys/bus/pci/drivers/bnx2/0000:05:00.0
/sys/bus/pci/drivers/bnx2/module
/sys/bus/pci/drivers/bnx2/uevent
/sys/bus/pci/drivers/bnx2/unbind
/sys/bus/pci/drivers/bnx2/bind
/sys/bus/pci/drivers/bnx2/new_id
/sys/bus/pci/drivers/bnx2/remove_id
/sys/module/bnx2
/sys/module/bnx2/holders
/sys/module/bnx2/version
/sys/module/bnx2/srcversion
/sys/module/bnx2/initstate
/sys/module/bnx2/refcnt
/sys/module/bnx2/sections
/sys/module/bnx2/sections/.note.gnu.build-id
/sys/module/bnx2/sections/.text
/sys/module/bnx2/sections/.exit.text
/sys/module/bnx2/sections/.devinit.text
/sys/module/bnx2/sections/.init.text
/sys/module/bnx2/sections/.devexit.text
/sys/module/bnx2/sections/.rodata
/sys/module/bnx2/sections/.rodata.str1.1
/sys/module/bnx2/sections/.smp_locks
/sys/module/bnx2/sections/__bug_table
/sys/module/bnx2/sections/.rodata.str1.8
/sys/module/bnx2/sections/__param
/sys/module/bnx2/sections/.devinit.rodata
/sys/module/bnx2/sections/.data
/sys/module/bnx2/sections/.devinit.data
/sys/module/bnx2/sections/.gnu.linkonce.this_module
/sys/module/bnx2/sections/.bss
/sys/module/bnx2/sections/.symtab
/sys/module/bnx2/sections/.strtab
/sys/module/bnx2/notes
/sys/module/bnx2/notes/.note.gnu.build-id
/sys/module/bnx2/drivers
/sys/module/bnx2/drivers/pci:bnx2
/lib/firmware/bnx2
/lib/firmware/bnx2/bnx2-rv2p-06-4.6.16.fw
/lib/firmware/bnx2/bnx2-mips-06-4.6.16.fw
/lib/firmware/bnx2/bnx2-rv2p-09-4.6.15.fw
/lib/firmware/bnx2/bnx2-mips-09-4.6.17.fw
/lib/modules/2.6.30.4-dgb/kernel/drivers/net/bnx2.ko
/lib/modules/2.6.24-19-server/kernel/drivers/net/bnx2.ko

But after boot I have again the same problem.

/etc/initramfs-tools/modules doesn't make reference to some module. I do not see that in Ubuntu Hardy firmware-bnx2 is a package, although am not sure either if it is necessary with kernel compiled from the sources.

With 2.6.30.4 it happens the same exactly.

Thanks for your reply.

Regards,
Daniel

[1] http://www.linux-kvm.org/page/Downloads

Fingerprint: BFB3 08D6 B4D1 31B2 72B9  29CE 6696 BF1B 14E6 1D37
Powered by Debian GNU/Linux Squeeze - Linux user #188.598

I think it's just luck; with kernel < 2.6.27 swapping behaviour will not be very good.

Please take it to netdev, perhaps they can help you with this.

error compiling committee.c: too many arguments to function

Hi Daniel!

Is bnx2.ko loaded by initramfs? Then the initramfs must contain the requested firmware file.

Another possibility is to set FIRMWARE_IN_KERNEL in the kernel config.

Bernhard

Hi Bernhard!

I've set in the kernel config:

CONFIG_FIRMWARE_IN_KERNEL=y

Although I don't see the asked file in initramfs. The file with full path is /lib/firmware/bnx2/bnx2-mips-06-4.6.16.fw:

# zcat /boot/initrd.img-2.6.30.4-dgb | cpio -t | grep bnx2
lib/modules/2.6.30.4-dgb/kernel/drivers/net/bnx2.ko
300859 blocks

Whereupon this variable is set to 'y' would not have to be sufficient?

<just_guessing>
1) Perhaps firmware must be in some other directory.
2) Perhaps there is no access to the FS in /lib at the moment that loads drivers.
</just_guessing>

Thanks for your reply.

Regards,
Daniel

Fingerprint: BFB3 08D6 B4D1 31B2 72B9  29CE 6696 BF1B 14E6 1D37
Powered by Debian GNU/Linux Squeeze - Linux user #188.598

Hi Daniel!

And what happened? Are you sure you recompiled and booted the new kernel with CONFIG_FIRMWARE_IN_KERNEL=y?

Obviously bnx2.ko is loaded from initramfs. At that time the root filesystem is not accessible. I expected CONFIG_FIRMWARE_IN_KERNEL=y is sufficient to solve the problem.

Here is what I would try:
- make sure the firmware is included in your kernel, or
- keep bnx2.ko from including in initramfs (unless you boot via network), or
- include /lib/firmware/bnx2/bnx2-mips-06-4.6.16.fw in initramfs

HTH
Bernhard

Hi, Bernhard!

According to the tests that I was doing, the CONFIG_FIRMWARE_IN_KERNEL=y option is not sufficient. Also it is necessary to use CONFIG_BNX2=y.

According to which I was reading, with the CONFIG_FIRMWARE_IN_KERNEL=y option, the driver is including in kernel as long as driver it is compiled in kernel and not in modular form. In order to obtain this it is necessary to use the CONFIG_BNX2=y option. This option is established following the path Device Drivers -> Network device support -> Ethernet (1000 Mbit) of the ncurses menu. Then, we marked with asterisk Broadcom NetXtremeII support here.

With this, the interfaces were configured during bootstrapping without problems:

[    2.805440] Broadcom NetXtreme II Gigabit Ethernet Driver bnx2 v2.0.1 (May 6, 2009)
[    2.805458] bnx2 0000:03:00.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16
[    3.410016] bnx2 0000:03:00.0: firmware: using built-in firmware bnx2/bnx2-mips-06-4.6.16.fw
[    3.410019] bnx2 0000:03:00.0: firmware: using built-in firmware bnx2/bnx2-rv2p-06-4.6.16.fw
[    3.410139] eth0: Broadcom NetXtreme II BCM5708 1000Base-T (B2) PCI-X 64-bit 133MHz found at mem f8000000, IRQ 16, node addr 00:21:5a:d5:dc:42
[    3.410190] bnx2 0000:05:00.0: PCI INT A -> GSI 17 (level, low) -> IRQ 17
[    4.020012] bnx2 0000:05:00.0: firmware: using built-in firmware bnx2/bnx2-mips-06-4.6.16.fw
[    4.020014] bnx2 0000:05:00.0: firmware: using built-in firmware bnx2/bnx2-rv2p-06-4.6.16.fw
[    4.020113] eth1: Broadcom NetXtreme II BCM5708 1000Base-T (B2) PCI-X 64-bit 133MHz found at mem fa000000, IRQ 17, node addr 00:21:5a:d5:dc:40

Thanks for your reply.

Regards,
Daniel

Fingerprint: BFB3 08D6 B4D1 31B2 72B9  29CE 6696 BF1B 14E6 1D37
Powered by Debian GNU/Linux Squeeze - Linux user #188.598