Overcommit problems with 2.6.12-rc4 (on AMD64)
(Please Cc me on answers, I don't follow LKML.)
Hi,
Suddenly one of our servers, a Dual Opteron with 2GB memory (running 32-bit userland, but 64-bit kernel) started to behave oddly:
imapd[31528]: segfault at 00000000fff00000 rip 00000000556a1a6d rsp 00000000ffffd394 error 4 imapd[31527]: segfault at 00000000fff00000 rip 00000000556a1a6d rsp 00000000ffffcbe4 error 4 sh[31530]: segfault at 00000000ffff7ff4 rip 000000005555e556 rsp 00000000ffff7ff8 error 6 sh[31531]: segfault at 00000000ffff7e5c rip 00000000555dc575 rsp 00000000ffff7e60 error 6 Unable to load interpreter /lib/ld-linux.so.2
Unable to load interpreter /lib/ld-linux.so.2
(ad infinitum)
It turned out it had some sort of memory problem:
Jun 2 11:56:02 cassarossa smbd[7171]: oplock_break: malloc fail for input buffer. Jun 2 11:56:02 cassarossa smbd[7171]: open_mode_check: FAILED when breaking oplock (3) on file login.bat, dev = 900, inode = 110665
This wasn't a RAM problem, as the machine has ECC RAM and we received no warnings from it. Also, we definitely had enough swap:
cassarossa:~# free
total used free shared buffers cached
Mem: 2058300 2041136 17164 0 39576 1601468
-/+ buffers/cache: 400092 1658208
Swap: 3903712 0 3903712It looks like somehow, the kernel couldn't really distinguish between memory used as cache and just "used". It couldn't even swapoff:
cassarossa:~# swapoff -a swapoff: /dev/sda5: Cannot allocate memory swapoff: /dev/sdf5: Cannot allocate memory
However, we run with vm.overcommit_memory=2, so we figured out it was worth a shot:
cassarossa:~# echo 0 > /proc/sys/vm/overcommit_memory
cassarossa:~# swapoff -a
cassarossa:~# swapon -a
cassarossa:~# free -m
total used free shared buffers cached
Mem: 2010 1993 16 0 39 1595
-/+ buffers/cache: 358 1651
Swap: 3812 0 3812Suddenly everything seems to be back to normal (ie. we could swapoff, and the programs stopped running out of memory; no changes in the cache used, though), and after a quick restart of services, everything is back to normal. So to me, it looks like vm.overcommit_memory=2 is broken, at least on AMD64. Any ideas why this would happen?
for the record:
cassarossa:~# uname -a Linux cassarossa 2.6.12-rc4 #1 SMP Fri May 13 18:49:40 CEST 2005 x86_64 unknown
No kernel patches except for a microscopic forward-port of the ELF fix from 2.6.11.9.
/* Steinar */
Signature · 1 line
Homepage: http://www.sesse.net/
Quoted 7 lines · Steinar H. Gunderson
imapd[31528]: segfault at
00000000fff00000rip00000000556a1a6drsp00000000ffffd394error 4 imapd[31527]: segfault at00000000fff00000rip00000000556a1a6drsp00000000ffffcbe4error 4 sh[31530]: segfault at00000000ffff7ff4rip000000005555e556rsp00000000ffff7ff8error 6 sh[31531]: segfault at00000000ffff7e5crip00000000555dc575rsp00000000ffff7e60error 6 Unable to load interpreter/lib/ld-linux.so.2
Unable to load interpreter/lib/ld-linux.so.2
(ad infinitum)
You ran out of address space
Quoted 5 lines
Suddenly everything seems to be back to normal (ie. we could swapoff, and the programs stopped running out of memory; no changes in the cache used, though), and after a quick restart of services, everything is back to normal. So to me, it looks like vm.overcommit_memory=2 is broken, at least on AMD64. Any ideas why this would happen?
vm.overcommit_memory=2 prevents the possibility of overcommitting - ie of address space being allocated to someone which is not used. Your swap allocation data is showing pages allocated not pages that could be allocated due to page faults. You need to look as the AS figures in /proc/meminfo to see the address space committed.
Basically it went back to "sane" because you said "ok I might get OOM but take the gamble", and since many programs allocate lots of space they never touch it worked.