Skip to contentSkip to conversation

Prevent OOM from killing init

PATCH [email protected] 11 participants 14 messages Mar 23, 2001 – Mar 25, 2001

lore
Share & tools
lore
#0 lore raw
Quoted 8 lines · Alan Cox
Quoted 4 lines

This is just an escape route in case everything else has failed.

Linux is unreliable.
That is bad.

Since your definition of reliability is a mathematical abstraction requiring infinite storage why don't you start by inventing infinitely large SDRAM chips, then get back to us ?

Ah, Alan,
I can see that you dislike seeing me say bad things about Linux. I dislike having to say them.

On the other hand, my definition of reliability does not require infinite storage. After all, earlier Unix flavours did not need an OOM killer either, and my editor was not killed under Unix V6 on 64k when I started some other process.

Linux is unreliable because a program can be killed at random, without warning, because of bugs in some other program. The old Unix guarantee that a program only crashes because of its own behaviour is lost. That is very sad.

What can one do? I need not tell you - you know better than I do. The main point is letting malloc fail when the memory cannot be guaranteed. There are various solutions for stack space, none of them very elegant, but all have in common that when we run out of stack space the program doing that gets SIGSEGV, and not some random other program. (And a well-written program could catch this SIGSEGV and do cleanup, preserving the integrity of its data base. Clearly one would want to guarantee a certain minimum stack space at fork time.)

Will this setup be very inefficient? I don't know. Perhaps. If my programs actually use 10 MB but have a guarantee for 200 MB then the rest of that memory is not wasted. But it can only be used for things that can be freed when needed, like inode and buffer cache.

But inefficient or not, I much prefer a system with guarantees, something that is reliable by default, above something that works well if you are lucky and fails at unpredictable moments.

Andries

#1 lore raw
Quoted 3 lines

infinite storage. After all, earlier Unix flavours did not need an OOM killer either, and my editor was not killed under Unix V6 on 64k when I started some other process.

You were lucky. Its quite possible for V6 to kill processes when you run out of swap

Quoted 2 lines

The old Unix guarantee that a program only crashes because of its own behaviour is lost. That is very sad.

No such guarantee ever existed. There are systems that had stuff like per user memory quotas but those were mostly much more mainframe oriented

Quoted 3 lines

200 MB then the rest of that memory is not wasted. But it can only be used for things that can be freed when needed, like inode and buffer cache.

No. You cannot free the inode and buffer cache arbitarily. You only have a probability - that puts you back at square 1.

Quoted 3 lines

But inefficient or not, I much prefer a system with guarantees, something that is reliable by default, above something that works well if you are lucky and fails at unpredictable moments.

malloc is merely an accounting exercise (actually its mostly mmap accounting). ptrace is the only quirk. Nobody feels its very important because nobody has implemented it.

Alan

#2 lore raw
Quoted 6 lines · Alan Cox
Quoted 3 lines

infinite storage. After all, earlier Unix flavours did not need an OOM killer either, and my editor was not killed under Unix V6 on 64k when I started some other process.

You were lucky. Its quite possible for V6 to kill processes when you run out of swap

Not lucky. I've used V6 - It would not start a process if the resources werent available (no overcommit). It was also a swap based system and not a page based system (PDP-11/45 1123+... supported both, but UNIX only used swapping because it was easy to swap a 64Kbyte process).

Quoted 5 lines
Quoted 2 lines

The old Unix guarantee that a program only crashes because of its own behaviour is lost. That is very sad.

No such guarantee ever existed. There are systems that had stuff like per user memory quotas but those were mostly much more mainframe oriented

Only the swapping based systems gave this guarantee. Even AT&T System V release 2 was swap based (M68020 systems).

Quoted 7 lines
Quoted 3 lines

200 MB then the rest of that memory is not wasted. But it can only be used for things that can be freed when needed, like inode and buffer cache.

No. You cannot free the inode and buffer cache arbitarily. You only have a probability - that puts you back at square 1.

Quoted 7 lines
Quoted 3 lines

But inefficient or not, I much prefer a system with guarantees, something that is reliable by default, above something that works well if you are lucky and fails at unpredictable moments.

malloc is merely an accounting exercise (actually its mostly mmap accounting). ptrace is the only quirk. Nobody feels its very important because nobody has implemented it.

Small correction - It was implemented, just not included in the standard kernel.

Check mailing lists around March-April of 2000. The patch was generated by Eduardo Horvath <[email protected]> for 2.3.99-pre3 and allowed the administrator to:

"Available virtual memory is calculated as the sum of all swap space as
well as free and reclaimable RAM, essentially the same value as used before. The kernel will now operate in 4 different modes depending on the value of sysctl_overcommit_memory:

1 Do accounting but do not prevent any allocations (old behavior)

0 Do accounting but only prevent individual allocations that exceed
total VM (old behavior)

-1 Do accounting and prevent a user from making the amount of
reserved memory exceed the total virtual memory.

-2 Same as above but also for root.

The default is set to -1 to allow root to essentially do whatever it wants. But then if someone's broken root you're in trouble anyway.

If the kernel itself requires memory it can allocate as much as it wants and can bring the system into an unsafe state (reserved > total).

Memory segments that are not COW, ZFOD or otherwise swap backed do not require reservation."

It was a limited implementation, but worked quite well in testing.

Signature · 5 lines
-------------------------------------------------------------------------
Jesse I Pollard, II
Email: [email protected]

Any opinions expressed are solely my own.
#3 lore raw
Quoted 1 line · Jesse Pollard

On Fri, 23 Mar 2001, Alan Cox wrote:

[ .... about non-overcommit .... ]

Quoted 1 line
Quoted 1 line

Nobody feels its very important because nobody has implemented it.

Enterprises use other systems because they have much better resource management than Linux -- adding non-overcommit wouldn't help them much. Desktop users, Linux newbies don't understand what's eager/early/non-overcommit vs lazy/late/overcommit memory management [just see these threads here if you aren't bored already enough ;)] and even if they do at last they don't have the ability to implement it. And between them, people are mostly fine with ulimit.

Quoted 2 lines

Small correction - It was implemented, just not included in the standard kernel.

Please note, adding optional non-overcommit also wouldn't help much without guaranteed/reserved resources [e.g. you are OOM -> appps, users complain, admin login in and BANG OOM killer just killed one of the jobs]. This was one of the reasons I made the reserved root memory patch [this is also the way other OS'es do]. Now just the different patches should be merged and write an OOM FAQ for users how to avoid, control, etc it].

Szaka

#4 lore raw
Quoted 14 lines

[ .... about non-overcommit .... ]

Quoted 1 line
Quoted 1 line

Nobody feels its very important because nobody has implemented it.

Enterprises use other systems because they have much better resource management than Linux -- adding non-overcommit wouldn't help them much. Desktop users, Linux newbies don't understand what's eager/early/non-overcommit vs lazy/late/overcommit memory management [just see these threads here if you aren't bored already enough ;)] and even if they do at last they don't have the ability to implement it. And between them, people are mostly fine with ulimit.

Quoted 2 lines

Small correction - It was implemented, just not included in the standard kernel.

Quoted 7 lines

Please note, adding optional non-overcommit also wouldn't help much without guaranteed/reserved resources [e.g. you are OOM -> appps, users complain, admin login in and BANG OOM killer just killed one of the jobs]. This was one of the reasons I made the reserved root memory patch [this is also the way other OS'es do]. Now just the different patches should be merged and write an OOM FAQ for users how to avoid, control, etc it].

I'm currently trying to apply the 2.3.99.whatever non-overcommit patch to 2.4.1 - decidedly nontrivial, lots of failed hunks, parts of the kernel have changed significantly even in this (fairly short) time.

--------------------------------------------------------------

from:     Jonathan "Chromatix" Morton
mail:     [email protected]  (not for attachments)
big-mail: [email protected]
uni-mail: [email protected]

The key to knowledge is not to rely on people to teach you it.

Get VNC Server for Macintosh from http://www.chromatix.uklinux.net/vnc/

-----BEGIN GEEK CODE BLOCK-----
Version 3.12
GCS$/E/S dpu(!) s:- a20 C+++ UL++ P L+++ E W+ N- o? K? w--- O-- M++$ V? PS PE- Y+ PGP++ t- 5- X- R !tv b++ DI+++ D G e+ h+ r++ y+(*) -----END GEEK CODE BLOCK-----

#5 lore raw
In reply to [email protected]

Please point me to an Operating System that runs on any commonly available platform and fits your requirements.
Nick

Quoted 51 lines · On Fri, 23 Mar 2001 [email protected] wrote:
Quoted 8 lines · Alan Cox
Quoted 4 lines

This is just an escape route in case everything else has failed.

Linux is unreliable.
That is bad.

Since your definition of reliability is a mathematical abstraction requiring infinite storage why don't you start by inventing infinitely large SDRAM chips, then get back to us ?

Ah, Alan,
I can see that you dislike seeing me say bad things about Linux. I dislike having to say them.

On the other hand, my definition of reliability does not require infinite storage. After all, earlier Unix flavours did not need an OOM killer either, and my editor was not killed under Unix V6 on 64k when I started some other process.

Linux is unreliable because a program can be killed at random, without warning, because of bugs in some other program. The old Unix guarantee that a program only crashes because of its own behaviour is lost. That is very sad.

What can one do? I need not tell you - you know better than I do. The main point is letting malloc fail when the memory cannot be guaranteed. There are various solutions for stack space, none of them very elegant, but all have in common that when we run out of stack space the program doing that gets SIGSEGV, and not some random other program. (And a well-written program could catch this SIGSEGV and do cleanup, preserving the integrity of its data base. Clearly one would want to guarantee a certain minimum stack space at fork time.)

Will this setup be very inefficient? I don't know. Perhaps. If my programs actually use 10 MB but have a guarantee for 200 MB then the rest of that memory is not wasted. But it can only be used for things that can be freed when needed, like inode and buffer cache.

But inefficient or not, I much prefer a system with guarantees, something that is reliable by default, above something that works well if you are lucky and fails at unpredictable moments.

Andries
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

#6 lore raw
Quoted 4 lines · [email protected] wrote:

Please point me to an Operating System that runs on any commonly available platform and fits your requirements.
Nick

You don't beleve me if I tell you: DOS extender and JVM (Java Virtual Machine)

#7 lore raw

The only thing out of that I don't belive is that it's a useable Operating System. I like your solution though. Thanks for actually comeing up with a useable solution instead of mindlessly ranting.
Nick

Quoted 14 lines · Martin Dalecki
Quoted 4 lines · [email protected] wrote:

Please point me to an Operating System that runs on any commonly available platform and fits your requirements.
Nick

You don't beleve me if I tell you: DOS extender and JVM (Java Virtual Machine)
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

#8 lore raw
In reply to Martin Dalecki
Quoted 2 lines

You don't beleve me if I tell you: DOS extender and JVM (Java Virtual Machine)

The JVM doesnt actually. The JVM will itself spontaenously explode in real life when out of memory. Maybe the JVM on a DOS extender 8)

#9 lore raw
Quoted 4 lines · Alan Cox
Quoted 3 lines

You don't beleve me if I tell you: DOS extender and JVM (Java Virtual Machine)

Quoted 8 lines

The JVM doesnt actually. The JVM will itself spontaenously explode in real life when out of memory. Maybe the JVM on a DOS extender 8)

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Back in the early nineties I was working with 18 developers on a Data General Aviion running DGUX. The system had only 16mb of memory and 600mb of disk. We were all continuously going thru the edit, compile, debug steps developing as large Computer Aided Dispatch System. Never did this system with its limited resources crash, or randomly start killing user or system processes.

My $.02.
Steve

#10 lore raw
Quoted 14 lines · Stephen E. Clark
Quoted 4 lines · Alan Cox
Quoted 3 lines

You don't beleve me if I tell you: DOS extender and JVM (Java Virtual Machine)

Quoted 8 lines

The JVM doesnt actually. The JVM will itself spontaenously explode in real life when out of memory. Maybe the JVM on a DOS extender 8)

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Quoted 6 lines

Back in the early nineties I was working with 18 developers on a Data General Aviion running DGUX. The system had only 16mb of memory and 600mb of disk. We were all continuously going thru the edit, compile, debug steps developing as large Computer Aided Dispatch System. Never did this system with its limited resources crash, or randomly start killing user or system processes.

What about the following (it is an estimate):

early nineties  -->  early eighties
18 developers   -->  18 developers
16mb of memory  -->   1 mb of memory
600 mb of disk  -->  70 mb of disk

Most current applications are so huge BLOATAGE that they should not deserve to be run just once. :-)
The kernel must try to cope with that and also with its own BLOATAGE.

Human nature is to eat what can be eaten, regardless if it is useful or not.

Quoted 1 line

My $.02.

What about 'My M$.02' in some decades. :)

Btw, 'decade' comes from Latin 'deca'=10 and dies=days (not sure for dies). As a result, it should have meant a period of 10 days instead of 10 years. It means a period of 10 days in French.

May-be, a knowledgeable person at this list has an explanation for this misinterpretation. Could it be due to the word 'decadent' that has a very different ethymology.

10 days is too short for getting decadent, but 10 years should be enough, no ? :-)

Quoted 1 line

Steve

Gérard.

#11 lore raw
In reply to [email protected]
Quoted 2 lines

The main point is letting malloc fail when the memory cannot be guaranteed.

If I read various things correctly, malloc() is supposed to fail as you would expect if /proc/sys/vm/overcommit_memory is 0. This is the case on my RH 6.2 box, dunno about yours. I can write a simple test program which simply allocates tons of memory if you like...

...and I did. It filled up my physical and swap memory, and got killed by the OOM handler before malloc() failed, even though overcommit_memory was set to 0.

*****BAD!*****

Here's my test program and output (on a Duron with 256M physical and 250M swap):

[chromi@beryllium compsci]$ cat make_mem.c
#include <stdio.h>
#include <stdlib.h>

int main(void)
{
/* Allocate tons of RAM, print out how far, we get, and exit when we
malloc() fails.
* We also access each page we allocate, to ensure we really are getting
the memory we reserve.
* If we are killed by SIGSEGV or by OOM instead of malloc() failing, the
VM system is broken.
*/

  char *p;
  unsigned long pages = 0;

  while(1) {
    p = malloc(1024);
    if(!p)
      break;
    *p = 1;
    pages++;
    printf("%lu K\r", pages);
  }

  printf("\n*** malloc() failed!\n");

return 0;
}
[chromi@beryllium compsci]$ gcc -O -Wall -o make_mem make_mem.c [chromi@beryllium compsci]$ ./make_mem
493625 KKilled
[chromi@beryllium compsci]$

--------------------------------------------------------------

from:     Jonathan "Chromatix" Morton
mail:     [email protected]  (not for attachments)
big-mail: [email protected]
uni-mail: [email protected]

The key to knowledge is not to rely on people to teach you it.

Get VNC Server for Macintosh from http://www.chromatix.uklinux.net/vnc/

-----BEGIN GEEK CODE BLOCK-----
Version 3.12
GCS$/E/S dpu(!) s:- a20 C+++ UL++ P L+++ E W+ N- o? K? w--- O-- M++$ V? PS PE- Y+ PGP++ t- 5- X- R !tv b++ DI+++ D G e+ h+ r++ y+(*) -----END GEEK CODE BLOCK-----

#12 lore raw
Quoted 8 lines · Jonathan Morton
Quoted 2 lines

The main point is letting malloc fail when the memory cannot be guaranteed.

If I read various things correctly, malloc() is supposed to fail as you would expect if /proc/sys/vm/overcommit_memory is 0. This is the case on my RH 6.2 box, dunno about yours. I can write a simple test program which simply allocates tons of memory if you like...

Quoted 5 lines

...and I did. It filled up my physical and swap memory, and got killed by the OOM handler before malloc() failed, even though overcommit_memory was set to 0.

*****BAD!*****

Please search list archives, there are plenty of threads about overcommitment.

Have a look at the sources, that part is easy to read and you'll realize that /proc/sys/vm/overcommit_memory does not really enable / disable memory overcommitment: its closer to a sanity check to disallow absurdly sized requests, IIRC.

.TM.

Signature · 5 lines
      ____/  ____/   /
     /      /       /			Marco Colombo
    ___/  ___  /   /		      Technical Manager
   /          /   /			 ESI s.r.l.
 _____/ _____/  _/		       [email protected]
#13 lore raw
In reply to [email protected]
Quoted 7 lines · Alan Cox
Quoted 3 lines

infinite storage. After all, earlier Unix flavours did not need an OOM killer either, and my editor was not killed under Unix V6 on 64k when I started some other process.

You were lucky. Its quite possible for V6 to kill processes when you run out of swap

It was actually worse than that. Grab your copy of "Lions", and check lines 4375-4377 in function xswap(). A failure to allocate space in the swapmap caused a panic. Same problem in xalloc().

Tim

Signature · 4 lines
Tim Wright - [email protected] or [email protected] or [email protected]
IBM Linux Technology Center, Beaverton, Oregon
Interested in Linux scalability ? Look at http://lse.sourceforge.net/
"Nobody ever said I was charming, they said "Rimmer, you're a git!"" RD VI