KVM and the OOM-Killer

This is *NOT* a KVM issue, but may be worth adding into the FAQ...

We have a KVM host with 48Gb of RAM and run about 20 KVM clients on it. After some time - different time depending on the kernel version - the VM host kernel will start OOM-Killing the VM clients, even when there is lots of free RAM (>10Gb) and free SWAP (>34Gb).

This seems to be caused by the kernel running out of LOWMEM (memory below 1Gb) - because of the large amount of RAM a lot of LOWMEM (~400Mb) is used by the memory map (32 bytes per 4Kb page), add in the kernel itself and that leaves "only" about 460Mb of LOWMEM for kernel alloc.

This may not have been a problem, except Linux may also put cache blocks and user processes in LOWMEM - it seems this can then lead to a LOWMEM exhaust situation which triggers OOM-Killing even when there is LOADS of SWAP and HIGHMEM free.

Sadly, killing userland processes is not a good way to try and free LOWMEM, so what happens is a killing spree where by every process on the VM host gets killed (inc all the VMs, sysklogd, klogd, sshd, udevd etc).

This is very bad in 2.6.32.6, quite bad in 2.6.32.9, better (but still bad in) 2.6.31.12 - currently testing 2.6.33.3

See https://bugzilla.kernel.org/show_bug.cgi?id=15058

General advice seems to be, if you have more than 16Gb RAM then you should run the VM host 64bit.

We didn't see this issue on a server with 32Gb running the same set of VMs.

James

I'd go with 64-bit at 2GB and above. It's both faster and safer.

The lowmem load is about 0.5% of guest memory, so 48GB means 240MB lowmem allocated. Thin ice.

Since you can run a 64-bit kernel with your existing userspace, at least you have a simple upgrade path.

error compiling committee.c: too many arguments to function

safer, how? (apart from no lowmem exhaust).

On a different subject, the qemu documentation says a guest VM can only have 2Gb of memory - does this still apply when using a 64bit host O/S ?

We currently only have about 12Gb used by the VM guests, so not hitting that issue - the rest of HIGHMEM is host disk cache.

Our experience is that the bottle-neck on number of VM guests is disk i/o - with loads of memory we've pretty much eliminated reads, so that means disk writes.

Not sure I like the idea of running a 64bit user space kernel on top of a 32bit host, prefer to re-install.

Can I just replace my kernel with a 64bit one, or do I have to re-install the host O/S ?

James

Nothing apart from that (well, if you run nonpae you lose NX protection).

The documentation is outdated.

You can run 32-bit userspace with a 64-bit kernel, or reinstall, whichever you prefer.

I once upgraded a 32-bit Fedora install to 64-bit, but that takes some tinkering.

error compiling committee.c: too many arguments to function

You can just install a 64-bit kernel. For rpm based systems you have to "unpack" the rpm using rpm2cpio. The modules.dep file cannot be updated -- need to generate that elsewhere -- and mkinitrd needs to be modified to not try to strip modules (s,strip,true,).

That's all I had to do to plop a 64-bit kernel onto a 32-bit install.

David

There's an interesting posting from Linus which explains details:
http://www.realworldtech.com/forums/index.cfm?action=detail&id=78966&threadid=78766&roomid=2

HTH,
Johannes

It seems going to a 64 bit kernel is what you want, but I thought it
worth mentioning the available method to say "try not to OOM-kill *this* process":

	echo "-16" > /proc/<pid>/oom_adj

I go through some convolutions in starting KVM VMs so that this gets done automatically for them. It basically scores the process *way* down for consideration to be OOM-killed. Handy if you might accidentally run something on the KVM host that chews up all the available memory. It should then get killed instead of the VMs. You can also use '-17' to exclude the process from consideration entirely.
See <kernel source>/Documentation/filesystems/proc.txt for details.

- Athanasius = Athanasius(at)miggy.org / http://www.miggy.org/
                  Finger athan(at)fysh.org for PGP key
	   "And it's me who is my enemy. Me who beats me up.
Me who makes the monsters. Me who strips my confidence." Paula Cole - ME

Thanks for that - yes, I know about "oom_adj", but it doesn't (totally) work. "udevd" has a default of "-17" and it got killed anyway.

Also, the only thing this server runs is VMs so if they can't be killed oom-killer will just run through the everything else (syslogd, sshd, klogd, udevd, hald, agetty etc) - so on balance its a case of which is worse? Without those daemons the system can become inaccessible and could become unstable, so on balance it may be better to let it kill the VMs.

My current work-around is :-

sync; echo 3 > /proc/sys/vm/drop_caches

I run it once a week midday Sunday in a cron job. This drops all blocks from the disk cache, and free up pretty all of LOWMEM - which sounds terrible, but its not that bad as each VM has its own cache anyway.

After running it I get ~2000 blocks/sec read for 1 minute, then ~20 to 50 blocks / sec read for the next few hours - which in terms of our disk RAID is nearly nothing. My "normal" level of disk reads is zero.

BTW: I've had very good results with "-cpu host". Its reduced my context switching considerably - up to 50% on one server - and reduced disk i/o (not quite sure why!).

James

Have you looked at memory cgroups and using that with limits with VMs?

	Three Cheers,
	Balbir

The problem was *NOT* that my VMs exhausted all memory. I know that is what "normally" triggers oom-killer, but you have to understand this mine was a very different scenario, hence I wanted to bring it to people's attention. I had about 10Gb of *FREE* HIGH and 34GB of *FREE* SWAP when oom-killer was activated - yep, didn't make sense to me either. If you want to study the logs :-

https://bugzilla.kernel.org/show_bug.cgi?id=15058

Looks like the problem was LOWMEM exhaust that triggered oom-killer. Which is dumb, because it was cache that was exhausting LOWMEM, and killing userland processes isn't a great way to deal with that issue.

[My] VMs generally alloate all resource at start-up and that's it.

Committed_AS: 14345016 kB

I tried "vm.overcommit_memory=2" and that didn't help. On a 48Gb system oom-killer should NEVER be invoked with that kind of memory profile - Its a quirk of running a 32bit system with *so* much memory, and the way pre-2.6.33 handled LOWMEM.

We've now moved all VM guests onto one server in preparation for a re-install of the other with 64bit host O/S.

Tests with 2.6.33.3 (+ latest qemu) appear to show this issue is fixed in the latest kernel (I can see it has much improved LOWMEM management), but we've only been running it days, and it can take 3 to 4 weeks to trigger.

FYI: We run about 100 VM guests on 7 VM hosts in five data centres - mostly production, some development. We've been using KVM in a production environment for a while now - starting [in production] at about KVM-82 on 2.6.28 - our oldest live systems now are two on KVM-84 on 2.6.28.4 and they are rock solid (one gets more punishment than it deserves) - but they only have 16Gb, so aren't seeing LOWMEM exhaust cos their memory map is *so* much smaller.

James

I understand, You could potentially encapsulate all else - except your VM's in a small cgroup and frequently reclaim from there using the memory cgroup. If drop caches works for you, that is good too. I am surprised that cache allocations are causing lowmem exhaustion.

	Three Cheers,
	Balbir

I'm suirprised that LOWMEM exhaustion causes oom-killer (when HIGH & SWAP are half empty) but there you go, life is full of surprises ;)

Of course, it could be something else, but its definitely disk i/o that triggers the oom-killer and clearing the cache stops it from happening (2.6.31.12) - any alternative explain really really welcome.

James

A lot of this is being changed, but not yet committed. There are patches out there to deal with the lowmem issue. Meanwhile, do follow the suggestions on oom_adj and moving to 64 bit.

	Three Cheers,
	Balbir