Skip to contentSkip to conversation

tmpfs and the OOM killer

DISCUSSION Pedro 6 participants 15 messages Apr 11, 2007 – Apr 12, 2007

lore
Share & tools
lore

Pedro

author
#0 lore raw

After suffering some days from a not|mis configured tmpfs,

As the OOM killer is not Posix,

Better than to kill processes would be to resize tmpfs, to use tmpfs empty
space.

I'm using kernel 2.6.20.4. If someone ask I'll send a test application.

#1 lore raw
Quoted 6 lines · Pedro

After suffering some days from a not|mis configured tmpfs,

As the OOM killer is not Posix,

Better than to kill processes would be to resize tmpfs, to use tmpfs empty
space.

Will not work, because tmpfs does not use any memory for unused space. If you don't believe me, simply create a large file on your tmpfs, then check free memory, then remove the file and check free memory again.

So your problem is not caused by the empty space on tmpfs, but either by too much space used on tmpfs or by your application using too much memory.

Quoted 1 line

I'm using kernel 2.6.20.4. If someone ask I'll send a test application.

Not needed, the one-liner "main(){while(malloc(4096));}" is enough to trigger an OOM.

If you cannot control your application's memory usage, you'll have to finely tune the overcommit_ratio.

Regards,
Willy

#2 lore raw

Can someone describe the process of finding the best value to tune the overcommit_ratio to?

-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of Willy Tarreau
Sent: Wednesday, April 11, 2007 3:48 PM
To: Pedro
Cc: [email protected]
Subject: Re: tmpfs and the OOM killer

Quoted 5 lines · Pedro

After suffering some days from a not|mis configured tmpfs,

As the OOM killer is not Posix,

Better than to kill processes would be to resize tmpfs, to use tmpfs

empty

Quoted 1 line

space.

Will not work, because tmpfs does not use any memory for unused space. If
you don't believe me, simply create a large file on your tmpfs, then check
free memory, then remove the file and check free memory again.

So your problem is not caused by the empty space on tmpfs, but either by too much space used on tmpfs or by your application using too much memory.

Quoted 1 line

I'm using kernel 2.6.20.4. If someone ask I'll send a test

application.

Not needed, the one-liner "main(){while(malloc(4096));}" is enough to trigger an OOM.

If you cannot control your application's memory usage, you'll have to finely
tune the overcommit_ratio.

Regards,
Willy

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

author
#3 lore raw
In reply to Willy Tarreau
Quoted 15 lines · Willy Tarreau
Quoted 6 lines · Pedro

After suffering some days from a not|mis configured tmpfs,

As the OOM killer is not Posix,

Better than to kill processes would be to resize tmpfs, to use tmpfs
empty space.

Will not work, because tmpfs does not use any memory for unused space. If you don't believe me, simply create a large file on your tmpfs, then check free memory, then remove the file and check free memory again.

So your problem is not caused by the empty space on tmpfs, but either by too much space used on tmpfs or by your application using too much memory.

...

Quoted 6 lines

If you cannot control your application's memory usage, you'll have to finely tune the overcommit_ratio.

Regards,
Willy

You are right.

But now I have two questions:

1) Why is tmpfs total space fixed if at the check moment does not exist
sufficient memory?

2) How should an application be written to not be killed by OOM?

#4 lore raw
Quoted 1 line

2) How should an application be written to not be killed by OOM?

OOM isn't an application matter. The kernel has to choose between allowing overcommit on the basis it might run out of memory and have to kill stuff, or that it won't in which case an applicatio which correctly handles malloc() and similar failures will not be killed (unless it is out of space on a stack grow which is a C language flaw as you can't catch that event in C)

It's configured by /proc/sys/vm/overcommit_memory

0 - try and spot obviously dumb allocations
1 - anything goes
2 - strictly control resource commit

author
#5 lore raw
Quoted 9 lines · Alan Cox
Quoted 1 line

2) How should an application be written to not be killed by OOM?

OOM isn't an application matter. The kernel has to choose between allowing overcommit on the basis it might run out of memory and have to kill stuff, or that it won't in which case an applicatio which correctly handles malloc() and similar failures will not be killed (unless it is out of space on a stack grow which is a C language flaw as you can't catch that event in C)

Quoted 5 lines

It's configured by /proc/sys/vm/overcommit_memory

0 - try and spot obviously dumb allocations
1 - anything goes
2 - strictly control resource commit

I deduce that a fail-safe application must scanf overcommit_memory, warn
the user and waitpid.

#13 lore raw
In reply to Willy Tarreau
Quoted 12 lines · Willy Tarreau
Quoted 6 lines · Pedro

After suffering some days from a not|mis configured tmpfs,

As the OOM killer is not Posix,

Better than to kill processes would be to resize tmpfs, to use tmpfs empty
space.

Will not work, because tmpfs does not use any memory for unused space. If you don't believe me, simply create a large file on your tmpfs, then check free memory, then remove the file and check free memory again.

Quoted 7 lines

So your problem is not caused by the empty space on tmpfs, but either by too much space used on tmpfs or by your application using too much memory.

Quoted 1 line

I'm using kernel 2.6.20.4. If someone ask I'll send a test application.

Not needed, the one-liner "main(){while(malloc(4096));}" is enough to trigger an OOM.

No, that won't do anything, malloc happily returns NULL after a few seconds.

int main()

{
	while(1) {
		char *p = malloc(4096);
		*p = 1;
	}
}

This is more likely to trigger OOM, because it actually dirties the page.

Jan

#14 lore raw
Quoted 25 lines · Jan Engelhardt
Quoted 12 lines · Willy Tarreau
Quoted 6 lines · Pedro

After suffering some days from a not|mis configured tmpfs,

As the OOM killer is not Posix,

Better than to kill processes would be to resize tmpfs, to use tmpfs empty
space.

Will not work, because tmpfs does not use any memory for unused space. If you don't believe me, simply create a large file on your tmpfs, then check free memory, then remove the file and check free memory again.

Quoted 7 lines

So your problem is not caused by the empty space on tmpfs, but either by too much space used on tmpfs or by your application using too much memory.

Quoted 1 line

I'm using kernel 2.6.20.4. If someone ask I'll send a test application.

Not needed, the one-liner "main(){while(malloc(4096));}" is enough to trigger an OOM.

No, that won't do anything, malloc happily returns NULL after a few seconds.

int main()

Quoted 8 lines
{
	while(1) {
		char *p = malloc(4096);
		*p = 1;
	}
}

This is more likely to trigger OOM, because it actually dirties the page.

Yes, you're right, and that's indeed what my "freemem" program does ;-)

Willy

#6 lore raw
In reply to Pedro
Quoted 6 lines
Quoted 3 lines

0 - try and spot obviously dumb allocations
1 - anything goes
2 - strictly control resource commit

I deduce that a fail-safe application must scanf overcommit_memory, warn
the user and waitpid.

If you are building a fail safe system you need to look a bit beyond out of memory handling settings - power supplies, failover, fault tolerance requirements, error detection (eg ECC ram), raid arrays over non-electrically conductive links etc.

Alan

#8 lore raw
In reply to Pedro
Quoted 7 lines · Pedro
Quoted 7 lines

OOM isn't an application matter. The kernel has to choose between allowing overcommit on the basis it might run out of memory and have to kill stuff, or that it won't in which case an applicatio which correctly handles malloc() and similar failures will not be killed (unless it is out of space on a stack grow which is a C language flaw as you can't catch that event in C)

Quoted 8 lines
Quoted 5 lines

It's configured by /proc/sys/vm/overcommit_memory

0 - try and spot obviously dumb allocations
1 - anything goes
2 - strictly control resource commit

I deduce that a fail-safe application must scanf overcommit_memory, warn
the user and waitpid.

If a fail-safe applicaion is running on a system which is that close to the edge in terms of available physical memory and swap, it's not likely going to be in deep trouble anyway. Even if you disable the OOM killer, now random malloc()'s will start returning NULL because your system doesn't have enough memory. Do you have intelligent error handling and recovery mechanisms for every single malloc() failure? Also, the machine will likely be thrashing so badly that any service level performance guarantees that the application might have will probably be totally trashed.

- Ted

author
#9 lore raw
Quoted 4 lines · Theodore Tso

likely going to be in deep trouble anyway. Even if you disable the OOM killer, now random malloc()'s will start returning NULL because your system doesn't have enough memory. Do you have intelligent error handling and recovery mechanisms for every single malloc() failure?

When malloc return NULL, the process may tell the user ENOMEM. When OOM kill the process, the user claim the program sometimes die.

author
#7 lore raw
In reply to Alan Cox
Quoted 2 lines · I wrote:
Quoted 2 lines

I deduce that a fail-safe application must scanf overcommit_memory,
warn the user and waitpid.

Quoted 4 lines · Alan Cox

If you are building a fail safe system you need to look a bit beyond out of memory handling settings - power supplies, failover, fault tolerance requirements, error detection (eg ECC ram), raid arrays over non-electrically conductive links etc.

If you lived here, you'ld be worried about seismic-electric impact on your
HDD and data ;) So SIGKILL is not the hardest problem to be solved.

Maybe it is time to write an avoidoomd.

#10 lore raw
In reply to Pedro
Quoted 8 lines · Pedro
Quoted 4 lines · Theodore Tso

likely going to be in deep trouble anyway. Even if you disable the OOM killer, now random malloc()'s will start returning NULL because your system doesn't have enough memory. Do you have intelligent error handling and recovery mechanisms for every single malloc() failure?

When malloc return NULL, the process may tell the user ENOMEM. When OOM kill the process, the user claim the program sometimes die.

Then use overcommit=2. The default overcommit mode is a convenience provided to allow poorly designed applications run even when they pretend they need gigs of RAM when they only use a few tens of megs.

If your application correctly handles malloc(), simply switch overcommit to 2 and let the system refuse to allocate memory when none is available, then your application will be aware of this by the NULL result to malloc() calls. It is a normal behaviour.

I do have appliances which run perfectly controlled software with overcommit_mode=2 and overcommit_ratio around 70% and without any swap, and they work like a charm. It just requires some finer grained tuning on your side. I don't see what the problem is here. You know the app, you know how much RAM you want to allocate to it, you know how much you want to keep free. Then say this to the system.

BTW, ulimit -v is your friend here too, and does not require to be root.

Regards,
Willy

#11 lore raw

I have noticed that with overcommit_memory=2 and overcommit_ratio=100, my system cannot leverage as much ram as it could if it was configured for overcommit_memory=0.

Is this because when overcommit_memory=2, anything that mallocs memory but doesn't touch that memory is counted as used memory? I see a value in /proc/meminfo called Commited_AS: and it seems to reflect what has been malloced in the system but not necessarily touched. Is this true?

Cheers,
Tony

-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of Willy Tarreau
Sent: Thursday, April 12, 2007 4:32 PM
To: Pedro
Cc: [email protected]
Subject: Re: tmpfs and the OOM killer

Quoted 4 lines · Pedro
Quoted 3 lines · Theodore Tso

likely going to be in deep trouble anyway. Even if you disable the OOM killer, now random malloc()'s will start returning NULL because your system doesn't have enough memory. Do you have intelligent

error

Quoted 4 lines
Quoted 1 line

handling and recovery mechanisms for every single malloc() failure?

When malloc return NULL, the process may tell the user ENOMEM. When OOM kill the process, the user claim the program sometimes die.

Then use overcommit=2. The default overcommit mode is a convenience provided
to allow poorly designed applications run even when they pretend they need
gigs of RAM when they only use a few tens of megs.

If your application correctly handles malloc(), simply switch overcommit to 2 and let the system refuse to allocate memory when none is available,
then your application will be aware of this by the NULL result to malloc()
calls. It is a normal behaviour.

I do have appliances which run perfectly controlled software with overcommit_mode=2 and overcommit_ratio around 70% and without any swap, and they work like a charm. It just requires some finer grained tuning on your side. I don't see what the problem is here. You know the app, you know how much RAM you want to allocate to it, you know how much you want to keep free. Then say this to the system.

BTW, ulimit -v is your friend here too, and does not require to be root.

Regards,
Willy

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

#12 lore raw
Quoted 6 lines · Tony

I have noticed that with overcommit_memory=2 and overcommit_ratio=100, my system cannot leverage as much ram as it could if it was configured for overcommit_memory=0.

Is this because when overcommit_memory=2, anything that mallocs memory but doesn't touch that memory is counted as used memory?

Most probably, yes. This proves that your system may endup doing OOM in mode 0 if one of your applications suddenly decided to use all the memory it has allocated.

Quoted 3 lines

I see a value
in /proc/meminfo called Commited_AS: and it seems to reflect what has been malloced in the system but not necessarily touched. Is this true?

Yes (at least I do think so).

Regards,
Willy