Skip to contentSkip to conversation

[2.4.26] overcommit_memory documentation clarification

DISCUSSION Vincent Lefevre 7 participants 10 messages May 9, 2004 – May 27, 2004

lore
Share & tools
lore
#0 lore raw

Hi,

The documentation of overcommit_memory in Documentation/sysctl/vm.txt should be clarified, as with the following simple program, malloc() never returns 0 on an official 2.4.26 kernel, even if overcommit_memory has been set to 0. Running it has the effect of having random processes killed, and eventually this process itself.

/* $Id: malloc.c 2753 2004-03-16 15:23:09Z lefevre $
*
* malloc() test
*/

#include <stdio.h>
#include <stdlib.h>
#include <string.h>

#define ONEMB 1048576

int main (void)
{
char *p;
int i;

for (i = 1; (p = malloc(ONEMB)) != NULL; i++)

    {
      printf ("Got %d MB\n", i);
      memset (p, 0, ONEMB);
    }

printf ("malloc() failed - OK\n");
return 0;
}

(After some discussions, it appears not to be a bug in the glibc
<http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=247300>.)

Signature · 4 lines
Vincent Lefèvre <[email protected]> - Web: <http://www.vinc17.org/>
100% validated (X)HTML - Acorn / RISC OS / ARM, free software, YP17,
Championnat International des Jeux Mathématiques et Logiques, etc.
Work: CR INRIA - computer arithmetic / SPACES project at LORIA
#1 lore raw
Quoted 7 lines · Vincent Lefevre

Hi,

The documentation of overcommit_memory in Documentation/sysctl/vm.txt should be clarified, as with the following simple program, malloc() never returns 0 on an official 2.4.26 kernel, even if overcommit_memory has been set to 0. Running it has the effect of having random processes killed, and eventually this process itself.

What made you think that malloc would return 0 if the system was "out of memory??" Malloc will return NULL, which is not 0 BTW, if you are out of address-space or have corrupted it by writing past a previous allocation. Malloc's return value is a void *. It should be compared against NULL, not zero.

When malloc() needs new "memory". It just asks the kernel to set the new break address or, in the case of mmap() mallocs, asks to extend a mapped region. Until somebody actually uses those regions, you haven't used any memory. So there is no way for malloc() to "know" ahead of time.

If you run a malloc() bomb from the root account you should end up killing off a lot of processes. If you run it from a normal user account, and you have set the user's resource quotas properly, only the user should get into trouble.

Cheers,
Dick Johnson
Penguin : Linux version 2.4.26 on an i686 machine (5557.45 BogoMips).
Note 96.31% of all statistics are fiction.

#2 lore raw
Quoted 5 lines · Richard B. Johnson

What made you think that malloc would return 0 if the system was "out of memory??" Malloc will return NULL, which is not 0 BTW, if you are out of address-space or have corrupted it by writing past a previous allocation. Malloc's return value is a void *. It should be compared against NULL, not zero.

You are wrong. They are the same value (this is required by the ISO C standard), i.e. both NULL == (void *) 0 and NULL == 0 must be true.

Quoted 5 lines

When malloc() needs new "memory". It just asks the kernel to set the new break address or, in the case of mmap() mallocs, asks to extend a mapped region. Until somebody actually uses those regions, you haven't used any memory. So there is no way for malloc() to "know" ahead of time.

Again, you are wrong. The goal of malloc is to reserve memory. This can be seen as used memory. If the implementation behaves differently, then it is broken.

Quoted 4 lines

If you run a malloc() bomb from the root account you should end up killing off a lot of processes. If you run it from a normal user account, and you have set the user's resource quotas properly, only the user should get into trouble.

The quotas are set properly (i.e. there are no quotas).

Signature · 4 lines
Vincent Lefèvre <[email protected]> - Web: <http://www.vinc17.org/>
100% validated (X)HTML - Acorn / RISC OS / ARM, free software, YP17,
Championnat International des Jeux Mathématiques et Logiques, etc.
Work: CR INRIA - computer arithmetic / SPACES project at LORIA
#3 lore raw
Quoted 1 line · Vincent

NULL == (void *) 0 and NULL == 0 must be true

Yes - NULL is compares equal to both (void *)0 and 0. No - not necessarily the _same_ value - one could be on a system with 32 bit ints, 64 bit pointers, for example.

Quoted 1 line

The goal of malloc is to reserve memory.

It's up to the kernel whether sbrk (used by malloc to obtain virtual address space) reserves memory or not.

Check out:
/proc/sys/vm/overcommit_memory Documentation/sysctl/vm.txt - overcommit_memory

Signature · 3 lines
                          I won't rest till it's the best ...
                          Programmer, Linux Scalability
                          Paul Jackson <[email protected]> 1.650.933.1373
#4 lore raw
Quoted 5 lines · Paul Jackson
Quoted 1 line · Vincent

NULL == (void *) 0 and NULL == 0 must be true

Yes - NULL is compares equal to both (void *)0 and 0. No - not necessarily the _same_ value - one could be on a system with 32 bit ints, 64 bit pointers, for example.

And so?

Quoted 3 lines
Quoted 1 line

The goal of malloc is to reserve memory.

It's up to the kernel whether sbrk (used by malloc to obtain virtual address space) reserves memory or not.

More old_mmap than brk (BTW, I forgot to say that this was on an x86 machine, I don't know if this matters...).

Quoted 3 lines

Check out:
/proc/sys/vm/overcommit_memory Documentation/sysctl/vm.txt - overcommit_memory

But the documentation is wrong (on an official 2.4.26 kernel). It seems that there is no way to get malloc() always return 0 when there isn't enough memory, even in simple cases (see my program posted in the first message).

Signature · 4 lines
Vincent Lefèvre <[email protected]> - Web: <http://www.vinc17.org/>
100% validated (X)HTML - Acorn / RISC OS / ARM, free software, YP17,
Championnat International des Jeux Mathématiques et Logiques, etc.
Work: CR INRIA - computer arithmetic / SPACES project at LORIA
#5 lore raw
Quoted 16 lines · Vincent Lefevre
Quoted 5 lines · Paul Jackson
Quoted 1 line · Vincent

NULL == (void *) 0 and NULL == 0 must be true

Yes - NULL is compares equal to both (void *)0 and 0. No - not necessarily the _same_ value - one could be on a system with 32 bit ints, 64 bit pointers, for example.

And so?

Quoted 3 lines
Quoted 1 line

The goal of malloc is to reserve memory.

It's up to the kernel whether sbrk (used by malloc to obtain virtual address space) reserves memory or not.

More old_mmap than brk (BTW, I forgot to say that this was on an x86 machine, I don't know if this matters...).

Quoted 8 lines
Quoted 3 lines

Check out:
/proc/sys/vm/overcommit_memory Documentation/sysctl/vm.txt - overcommit_memory

But the documentation is wrong (on an official 2.4.26 kernel). It seems that there is no way to get malloc() always return 0 when there isn't enough memory, even in simple cases (see my program posted in the first message).

Right.

We should or merge Alan's strict-overcommit patches (from RH's tree), or fix the documentation.

Marc-Christian Petersen has a patch to fixup the documentation.

#6 lore raw

On Thursday 27 May 2004 14:20, Marcelo Tosatti wrote:

Hi Marcelo,

Quoted 2 lines

We should or merge Alan's strict-overcommit patches (from RH's tree), or fix the documentation.

I vote for the strict-overcommit thing. Do you have that handy for your 2.4-bk?

Alan, prolly you? Or do we have to extract it from linux-2.4.21-selected-ac-bits.patch? ;)

ciao, Marc

#7 lore raw
Quoted 3 lines · Marc-Christian Petersen

On Thursday 27 May 2004 14:20, Marcelo Tosatti wrote:

Hi Marcelo,

Hi Marc,

Quoted 8 lines
Quoted 2 lines

We should or merge Alan's strict-overcommit patches (from RH's tree), or fix the documentation.

I vote for the strict-overcommit thing. Do you have that handy for your 2.4-bk?

Alan, prolly you? Or do we have to extract it from linux-2.4.21-selected-ac-bits.patch? ;)

Alan is busy with other things.

The strict overcommit fixes need to be extraced and tested.

Dave Jones told me about it the other day.

Still haven't found the time to download RH's srpm.

#8 lore raw
Quoted 11 lines · Marcelo Tosatti
Quoted 6 lines
Quoted 2 lines

We should or merge Alan's strict-overcommit patches (from RH's tree), or fix the documentation.

I vote for the strict-overcommit thing. Do you have that handy for your 2.4-bk?
Alan, prolly you? Or do we have to extract it from linux-2.4.21-selected-ac-bits.patch? ;)

Alan is busy with other things.
The strict overcommit fixes need to be extraced and tested. Dave Jones told me about it the other day.
Still haven't found the time to download RH's srpm.

The overcommit bits in the Fedora SRPM come directly from the 2.4.22-ac1 patch. You're better off just grabbing Bero's -pac patch. He has a 2.4.26 patch, which should save you rediffing, just chop out the relevant bits.

http://kernel.org/pub/linux/kernel/people/bero/2.4/2.4.27/

Dave

#9 lore raw
In reply to Marcelo Tosatti
Quoted 1 line · Marcelo Tosatti

The strict overcommit fixes need to be extraced and tested.

Bero's tree has them. If you merge it it is probably good to make the small changes so it works like 2.6