Skip to contentSkip to conversation

Overcommit problems with 2.6.12-rc4 (on AMD64)

DISCUSSION Steinar H. Gunderson 2 participants 2 messages Jun 2, 2005 – Jun 7, 2005

lore
Share & tools
lore
#0 lore raw

(Please Cc me on answers, I don't follow LKML.)

Hi,

Suddenly one of our servers, a Dual Opteron with 2GB memory (running 32-bit userland, but 64-bit kernel) started to behave oddly:

imapd[31528]: segfault at 00000000fff00000 rip 00000000556a1a6d rsp 00000000ffffd394 error 4 imapd[31527]: segfault at 00000000fff00000 rip 00000000556a1a6d rsp 00000000ffffcbe4 error 4 sh[31530]: segfault at 00000000ffff7ff4 rip 000000005555e556 rsp 00000000ffff7ff8 error 6 sh[31531]: segfault at 00000000ffff7e5c rip 00000000555dc575 rsp 00000000ffff7e60 error 6 Unable to load interpreter /lib/ld-linux.so.2
Unable to load interpreter /lib/ld-linux.so.2
(ad infinitum)

It turned out it had some sort of memory problem:

  Jun  2 11:56:02 cassarossa smbd[7171]: oplock_break: malloc fail for input buffer. 
  Jun  2 11:56:02 cassarossa smbd[7171]: open_mode_check: FAILED when breaking oplock (3) on file login.bat, dev = 900, inode = 110665 

This wasn't a RAM problem, as the machine has ECC RAM and we received no warnings from it. Also, we definitely had enough swap:

  cassarossa:~# free
               total       used       free     shared    buffers     cached
  Mem:       2058300    2041136      17164          0      39576    1601468
  -/+ buffers/cache:     400092    1658208
  Swap:      3903712          0    3903712

It looks like somehow, the kernel couldn't really distinguish between memory used as cache and just "used". It couldn't even swapoff:

  cassarossa:~# swapoff -a
  swapoff: /dev/sda5: Cannot allocate memory
  swapoff: /dev/sdf5: Cannot allocate memory

However, we run with vm.overcommit_memory=2, so we figured out it was worth a shot:

  cassarossa:~# echo 0 > /proc/sys/vm/overcommit_memory 
  cassarossa:~# swapoff -a
  cassarossa:~# swapon -a 
  cassarossa:~# free -m
               total       used       free     shared    buffers     cached
  Mem:          2010       1993         16          0         39       1595
  -/+ buffers/cache:        358       1651
  Swap:         3812          0       3812

Suddenly everything seems to be back to normal (ie. we could swapoff, and the programs stopped running out of memory; no changes in the cache used, though), and after a quick restart of services, everything is back to normal. So to me, it looks like vm.overcommit_memory=2 is broken, at least on AMD64. Any ideas why this would happen?

for the record:

  cassarossa:~# uname -a
  Linux cassarossa 2.6.12-rc4 #1 SMP Fri May 13 18:49:40 CEST 2005 x86_64 unknown

No kernel patches except for a microscopic forward-port of the ELF fix from 2.6.11.9.

/* Steinar */

Signature · 1 line
Homepage: http://www.sesse.net/
#1 lore raw
Quoted 7 lines · Steinar H. Gunderson

imapd[31528]: segfault at 00000000fff00000 rip 00000000556a1a6d rsp 00000000ffffd394 error 4 imapd[31527]: segfault at 00000000fff00000 rip 00000000556a1a6d rsp 00000000ffffcbe4 error 4 sh[31530]: segfault at 00000000ffff7ff4 rip 000000005555e556 rsp 00000000ffff7ff8 error 6 sh[31531]: segfault at 00000000ffff7e5c rip 00000000555dc575 rsp 00000000ffff7e60 error 6 Unable to load interpreter /lib/ld-linux.so.2
Unable to load interpreter /lib/ld-linux.so.2
(ad infinitum)

You ran out of address space

Quoted 5 lines

Suddenly everything seems to be back to normal (ie. we could swapoff, and the programs stopped running out of memory; no changes in the cache used, though), and after a quick restart of services, everything is back to normal. So to me, it looks like vm.overcommit_memory=2 is broken, at least on AMD64. Any ideas why this would happen?

vm.overcommit_memory=2 prevents the possibility of overcommitting - ie of address space being allocated to someone which is not used. Your swap allocation data is showing pages allocated not pages that could be allocated due to page faults. You need to look as the AS figures in /proc/meminfo to see the address space committed.

Basically it went back to "sane" because you said "ok I might get OOM but take the gamble", and since many programs allocate lots of space they never touch it worked.