tmpfs and the OOM killer
After suffering some days from a not|mis configured tmpfs,
As the OOM killer is not Posix,
Better than to kill processes would be to resize tmpfs, to use tmpfs empty
space.
I'm using kernel 2.6.20.4. If someone ask I'll send a test application.
Quoted 6 lines · Pedro
After suffering some days from a not|mis configured tmpfs,
As the OOM killer is not Posix,
Better than to kill processes would be to resize tmpfs, to use tmpfs empty
space.
Will not work, because tmpfs does not use any memory for unused space. If you don't believe me, simply create a large file on your tmpfs, then check free memory, then remove the file and check free memory again.
So your problem is not caused by the empty space on tmpfs, but either by too much space used on tmpfs or by your application using too much memory.
Quoted 1 line
I'm using kernel 2.6.20.4. If someone ask I'll send a test application.
Not needed, the one-liner "main(){while(malloc(4096));}" is enough to trigger an OOM.
If you cannot control your application's memory usage, you'll have to finely tune the overcommit_ratio.
Regards,
Willy
Can someone describe the process of finding the best value to tune the overcommit_ratio to?
-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of Willy Tarreau
Sent: Wednesday, April 11, 2007 3:48 PM
To: Pedro
Cc: [email protected]
Subject: Re: tmpfs and the OOM killer
Quoted 5 lines · Pedro
After suffering some days from a not|mis configured tmpfs,
As the OOM killer is not Posix,
Better than to kill processes would be to resize tmpfs, to use tmpfs
empty
Quoted 1 line
space.
Will not work, because tmpfs does not use any memory for unused space. If
you don't believe me, simply create a large file on your tmpfs, then check
free memory, then remove the file and check free memory again.
So your problem is not caused by the empty space on tmpfs, but either by too much space used on tmpfs or by your application using too much memory.
Quoted 1 line
I'm using kernel 2.6.20.4. If someone ask I'll send a test
application.
Not needed, the one-liner "main(){while(malloc(4096));}" is enough to trigger an OOM.
If you cannot control your application's memory usage, you'll have to finely
tune the overcommit_ratio.
Regards,
Willy
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Quoted 15 lines · Willy Tarreau
Quoted 6 lines · Pedro
After suffering some days from a not|mis configured tmpfs,
As the OOM killer is not Posix,
Better than to kill processes would be to resize tmpfs, to use tmpfs
empty space.Will not work, because tmpfs does not use any memory for unused space. If you don't believe me, simply create a large file on your tmpfs, then check free memory, then remove the file and check free memory again.
So your problem is not caused by the empty space on tmpfs, but either by too much space used on tmpfs or by your application using too much memory.
...
Quoted 6 lines
If you cannot control your application's memory usage, you'll have to finely tune the
overcommit_ratio.Regards,
Willy
You are right.
But now I have two questions:
1) Why is tmpfs total space fixed if at the check moment does not exist
sufficient memory?
2) How should an application be written to not be killed by OOM?
Quoted 1 line
2) How should an application be written to not be killed by OOM?
OOM isn't an application matter. The kernel has to choose between allowing overcommit on the basis it might run out of memory and have to kill stuff, or that it won't in which case an applicatio which correctly handles malloc() and similar failures will not be killed (unless it is out of space on a stack grow which is a C language flaw as you can't catch that event in C)
It's configured by /proc/sys/vm/overcommit_memory
0 - try and spot obviously dumb allocations
1 - anything goes
2 - strictly control resource commit
Quoted 9 lines · Alan Cox
Quoted 1 line
2) How should an application be written to not be killed by OOM?
OOM isn't an application matter. The kernel has to choose between allowing overcommit on the basis it might run out of memory and have to kill stuff, or that it won't in which case an applicatio which correctly handles
malloc()and similar failures will not be killed (unless it is out of space on a stack grow which is a C language flaw as you can't catch that event in C)
Quoted 5 lines
It's configured by
/proc/sys/vm/overcommit_memory0 - try and spot obviously dumb allocations
1 - anything goes
2 - strictly control resource commit
I deduce that a fail-safe application must scanf overcommit_memory, warn
the user and waitpid.
Quoted 6 lines
Quoted 3 lines
0 - try and spot obviously dumb allocations
1 - anything goes
2 - strictly control resource commitI deduce that a fail-safe application must scanf
overcommit_memory, warn
the user and waitpid.
If you are building a fail safe system you need to look a bit beyond out of memory handling settings - power supplies, failover, fault tolerance requirements, error detection (eg ECC ram), raid arrays over non-electrically conductive links etc.
Alan
Quoted 2 lines · I wrote:
Quoted 2 lines
I deduce that a fail-safe application must scanf
overcommit_memory,
warn the user and waitpid.
Quoted 4 lines · Alan Cox
If you are building a fail safe system you need to look a bit beyond out of memory handling settings - power supplies, failover, fault tolerance requirements, error detection (eg ECC ram), raid arrays over non-electrically conductive links etc.
If you lived here, you'ld be worried about seismic-electric impact on your
HDD and data ;) So SIGKILL is not the hardest problem to be solved.
Maybe it is time to write an avoidoomd.
Quoted 7 lines · Pedro
Quoted 7 lines
OOM isn't an application matter. The kernel has to choose between allowing overcommit on the basis it might run out of memory and have to kill stuff, or that it won't in which case an applicatio which correctly handles
malloc()and similar failures will not be killed (unless it is out of space on a stack grow which is a C language flaw as you can't catch that event in C)
Quoted 8 lines
Quoted 5 lines
It's configured by
/proc/sys/vm/overcommit_memory0 - try and spot obviously dumb allocations
1 - anything goes
2 - strictly control resource commitI deduce that a fail-safe application must scanf
overcommit_memory, warn
the user and waitpid.
If a fail-safe applicaion is running on a system which is that close to the edge in terms of available physical memory and swap, it's not likely going to be in deep trouble anyway. Even if you disable the OOM killer, now random malloc()'s will start returning NULL because your system doesn't have enough memory. Do you have intelligent error handling and recovery mechanisms for every single malloc() failure? Also, the machine will likely be thrashing so badly that any service level performance guarantees that the application might have will probably be totally trashed.
- Ted
Quoted 4 lines · Theodore Tso
likely going to be in deep trouble anyway. Even if you disable the OOM killer, now random malloc()'s will start returning NULL because your system doesn't have enough memory. Do you have intelligent error handling and recovery mechanisms for every single
malloc()failure?
When malloc return NULL, the process may tell the user ENOMEM. When OOM kill the process, the user claim the program sometimes die.
Quoted 8 lines · Pedro
Quoted 4 lines · Theodore Tso
likely going to be in deep trouble anyway. Even if you disable the OOM killer, now random malloc()'s will start returning NULL because your system doesn't have enough memory. Do you have intelligent error handling and recovery mechanisms for every single
malloc()failure?When malloc return NULL, the process may tell the user ENOMEM. When OOM kill the process, the user claim the program sometimes die.
Then use overcommit=2. The default overcommit mode is a convenience provided to allow poorly designed applications run even when they pretend they need gigs of RAM when they only use a few tens of megs.
If your application correctly handles malloc(), simply switch overcommit to 2 and let the system refuse to allocate memory when none is available, then your application will be aware of this by the NULL result to malloc() calls. It is a normal behaviour.
I do have appliances which run perfectly controlled software with overcommit_mode=2 and overcommit_ratio around 70% and without any swap, and they work like a charm. It just requires some finer grained tuning on your side. I don't see what the problem is here. You know the app, you know how much RAM you want to allocate to it, you know how much you want to keep free. Then say this to the system.
BTW, ulimit -v is your friend here too, and does not require to be root.
Regards,
Willy
I have noticed that with overcommit_memory=2 and overcommit_ratio=100, my system cannot leverage as much ram as it could if it was configured for overcommit_memory=0.
Is this because when overcommit_memory=2, anything that mallocs memory but doesn't touch that memory is counted as used memory? I see a value in /proc/meminfo called Commited_AS: and it seems to reflect what has been malloced in the system but not necessarily touched. Is this true?
Cheers,
Tony
-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of Willy Tarreau
Sent: Thursday, April 12, 2007 4:32 PM
To: Pedro
Cc: [email protected]
Subject: Re: tmpfs and the OOM killer
Quoted 4 lines · Pedro
Quoted 3 lines · Theodore Tso
likely going to be in deep trouble anyway. Even if you disable the OOM killer, now random malloc()'s will start returning NULL because your system doesn't have enough memory. Do you have intelligent
error
Quoted 4 lines
Quoted 1 line
handling and recovery mechanisms for every single
malloc()failure?When malloc return NULL, the process may tell the user ENOMEM. When OOM kill the process, the user claim the program sometimes die.
Then use overcommit=2. The default overcommit mode is a convenience provided
to allow poorly designed applications run even when they pretend they need
gigs of RAM when they only use a few tens of megs.
If your application correctly handles malloc(), simply switch overcommit to 2 and let the system refuse to allocate memory when none is available,
then your application will be aware of this by the NULL result to malloc()
calls. It is a normal behaviour.
I do have appliances which run perfectly controlled software with overcommit_mode=2 and overcommit_ratio around 70% and without any swap, and they work like a charm. It just requires some finer grained tuning on your side. I don't see what the problem is here. You know the app, you know how much RAM you want to allocate to it, you know how much you want to keep free. Then say this to the system.
BTW, ulimit -v is your friend here too, and does not require to be root.
Regards,
Willy
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Quoted 6 lines · Tony
I have noticed that with overcommit_memory=2 and overcommit_ratio=100, my system cannot leverage as much ram as it could if it was configured for overcommit_memory=0.
Is this because when overcommit_memory=2, anything that mallocs memory but doesn't touch that memory is counted as used memory?
Most probably, yes. This proves that your system may endup doing OOM in mode 0 if one of your applications suddenly decided to use all the memory it has allocated.
Quoted 3 lines
I see a value
in/proc/meminfocalledCommited_AS: and it seems to reflect what has been malloced in the system but not necessarily touched. Is this true?
Yes (at least I do think so).
Regards,
Willy
Quoted 12 lines · Willy Tarreau
Quoted 6 lines · Pedro
After suffering some days from a not|mis configured tmpfs,
As the OOM killer is not Posix,
Better than to kill processes would be to resize tmpfs, to use tmpfs empty
space.Will not work, because tmpfs does not use any memory for unused space. If you don't believe me, simply create a large file on your tmpfs, then check free memory, then remove the file and check free memory again.
Quoted 7 lines
So your problem is not caused by the empty space on tmpfs, but either by too much space used on tmpfs or by your application using too much memory.
Quoted 1 line
I'm using kernel 2.6.20.4. If someone ask I'll send a test application.
Not needed, the one-liner "main(){while(malloc(4096));}" is enough to trigger an OOM.
No, that won't do anything, malloc happily returns NULL after a few seconds.
int main()
{
while(1) {
char *p = malloc(4096);
*p = 1;
}
}This is more likely to trigger OOM, because it actually dirties the page.
Jan
Quoted 25 lines · Jan Engelhardt
Quoted 12 lines · Willy Tarreau
Quoted 6 lines · Pedro
After suffering some days from a not|mis configured tmpfs,
As the OOM killer is not Posix,
Better than to kill processes would be to resize tmpfs, to use tmpfs empty
space.Will not work, because tmpfs does not use any memory for unused space. If you don't believe me, simply create a large file on your tmpfs, then check free memory, then remove the file and check free memory again.
Quoted 7 lines
So your problem is not caused by the empty space on tmpfs, but either by too much space used on tmpfs or by your application using too much memory.
Quoted 1 line
I'm using kernel 2.6.20.4. If someone ask I'll send a test application.
Not needed, the one-liner "main(){while(malloc(4096));}" is enough to trigger an OOM.
No, that won't do anything, malloc happily returns NULL after a few seconds.
int
main()
Quoted 8 lines
{ while(1) { char *p = malloc(4096); *p = 1; } }This is more likely to trigger OOM, because it actually dirties the page.
Yes, you're right, and that's indeed what my "freemem" program does ;-)
Willy