VM nuisance

Is there anything measurably useful in any -ac or -pre patches after 2.4.7 that helps or fixes the blasted out-of-memory-but-let's-go-fsck -ourselves-for-a-few-hours?

I was very close to hitting the reset button and losing a lot of important information because of this. I accidently got too close to the edge of memory (~6megs free) and the kernel went into FMM (fsck myself mode)...i.e. spin mightily looking for memory and going noplace whilst ignoring it's little buddy the OOM handler.

Again, it doesn't matter if I have swap or not, if I get within ~6 megs of the end of memory, the kernel goes FMM. I've tested with and without swap. And _please_ don't tell me "just add more swap". That's ludicruous and isn't solving the problem, it's covering up a symptom.

</rant>

So, is there anything useful or any personal/private patches I can try?

David

No. The problem is that whenever I change something to the OOM killer I get flamed.

Both by the people for whom the OOM killer kicks in too early and by the people for whom the OOM killer now doesn't kick in.

I haven't got the faintest idea how to come up with an OOM killer which does the right thing for everybody.

regards,

Rik
--
IA64: a worthy successor to i860.

http://www.surriel.com/ http://distro.conectiva.com/

Send all your spam to [email protected] (spam digging piggy)

maybe these are signs that the OOM killer just wasn't ready for 2.4 with the
VM that it has. For people hoping to use 2.4 as a server platform, they
should have the confidence to know what's going to happen when the OOM
situation occurs. If the current OOM handler cant give that confidence then
perhaps it should be removed and slated for 2.5. Simple way to beat the
flames anyway.

How about adding some sort of per-process priority (i.e. a la nice) which would determine the order in which they would be OOMed? Then we could safely run X with a kigh KillMe and Netscape with an even higher KillMe and we would probably avoid the something useing too much memory let's kill root's shell...

[i.e. if a lower KillMe proc runs out of memory we kill off the process with the highest KillMe using most mem and can safely give this mem to the proc which just ran out]

MaZe.

That's not the problem. The killing itself is going pretty well.

The problem is to decide WHEN to kill a process.

regards,

Rik
--
IA64: a worthy successor to i860.

http://www.surriel.com/ http://distro.conectiva.com/

Send all your spam to [email protected] (spam digging piggy)

Rik Said:

Would there be a way that you could have a proc interface to adjust the sensitivity of the OOM killer so that users could raise or lower the threshold that causes OOM killer activation? Hopefully you wouldn't get any flak for that unless users start blaming you for their own settings :-)

Most people would just use the standard setting that you provide...but others that felt the need could change it on their system.

Dan

Though it is not a complete solution for all cases(servers etc), could a SysReq combination be added that triggers OOM? I'm sure many people use SysReq-k on occasion to get a system out of endless swapping, I think having a SysReq key for OOM would be a great improvement.

Comments?
-Justin

I think it's a damn good idea. I'd actually begin coding that now, if I knew where to start. SysRQ has saved my life more than once -- i'm sure it would help all the other people who are having problems with randomized thrashing and stuff.

Kelsey Hudson [email protected]
Software Engineer
Compendium Technologies, Inc (619) 725-0771
---------------------------------------------------------------------------

Wouldn't it be just a matter of

case 'f': /* F -- oom handlder _f_ree memory */

	printk("Run OOM Handler\n");
	oom_kill();
	break;

in handle_sysrq in sysrq.c?

-Justin

This causes a nasty ass BUG() to get asserted. I can trigger the OOM killer normally, but if I trigger it using SysRq, it'll complain rather loudly and kill the interrupt handler. It appears to kill the proper process first, however.

If I get some time later on, I'll boot up my devel machine at home and get this thing all squared away. It looks as though some sort of lock or something is being held when it shouldnt be, and it's causing Bad Things(tm) to happen. Either way, I'll figure it out at home (or when I get back from vacation on Monday).

LMK what you think.

Kelsey Hudson [email protected]
Software Engineer
Compendium Technologies, Inc (619) 725-0771
---------------------------------------------------------------------------

maybe the concept is flawed? not saying that it is flawed necessarily, but i've often wondered why linux vm issues never seem to be solved in the years i've been reading linux-kernel.

if i understand the OOM killer correctly it's intended to make some sort of intelligent choice as to which application to shoot when the system is out of memory.

honestly, if applications can't stomach being shot then the bug is in the application, not in the kernel. vi can handle being shot because it does the right thing: it checkpoints state periodically. it's a simple model which any sane application could follow, and many do actually follow.

getting shot for OOM or getting shot because of power failure, or 2-bit RAM failure, or backhoe fade, or ... what's really the difference?

so why not just use the most simple OOM around: shoot the first app which can't get its page. app writers won't like it, and users won't like it until the app writers fix their bugs, but then nobody likes the current situation, so what's the difference?

maybe kernel support for making checkpointing easier would be a good way to advance the science? there certainly are tools which exist that do part of the problem already -- except for sockets and pipes and such interprocess elements it's pretty trivial to checkpoint. interprocess elements probably require some extra kernel support. network elements are where it really becomes challenging.

i would happily give up 10 to 20% system resources for checkpoint overhead if it meant that i'd be that much closer to a crashproof system. after a year the performance deficit would be made up by hardware improvements.

i know it's a big pill to swallow, but i've been impressed time and time again by the will of linux kernel hackers to just say "this is how it will be, because it is the only known way to perfection, deal."

-dean

As I understand it, the problem is not what to do when the system is out of memory. The problem is deciding what actually consititutes out of memory. IOW, if I have a 32 MB system with 512MB of swap, I could have 200MB of swap free and still have a system so far gone that it swaps more than it works. OTOH, I could have 400MB worth of idle daemons and other residents taking up swap with a perfectly usable system in RAM.

It would be nice if we could sack a few tasks in swap for awhile so stuff in RAM could get things done. Something like "if a page gets swapped out, it'll stay there for at least a minute" should work fine in normal conditions (since the page has probably been idle for awhile, anyway). Under memory pressure, the scheduler would pass by any tasks waiting on a page that was just swapped out.

Now that I've shown off how little I know about the current OOM design, I'll go sit and be quiet...
-- Brian

dean gaudet wrote:
[...]

It used to be like that. Unfortunately, the first app unable to get its page might very well be init, and then the entire system goes down in flames. You might as well kill the kernel at that point.

Fix that, and people start complaining that the X server goes, taking all X apps with it when killing one would suffice. Fix that, and you almost have today's OOM killer.

The real solution is to have enough memory for the task at hand. Failing
that, get so much swap space that people will be happy when the OOM killer kicks in and limits the trashing.

Helge Hafting

It's not necessarily needed so much and not even checkpointing using (optional) non-overcommiting. See e.g. the Solaris numbers for yourself,
http://www.google.com/search?q=cache:http://lists.openresources.com/NetBSD/tech-userlevel/msg00722.html
All big guys support this optionally, sometimes even in a finer graded level, like per process settable by developer overjudgable (sp?) by admin (AIX) or using virtual swap space where the degree of the memory overcommitment controllable (IRIX), etc. Linux had the "quasi" non-overcommiting [it didn't reserved the demanded memory, just checked if there is enough free(able) in user space allocation time] controllable by /proc/sys/vm/overcommit_memory in 2.2 but in 2.4 vm_enough_memory() overestimates freeable memory (in contrast to 2.2 kernels) so basically it's useless - this is one of the reasons why OOM so trendy topic recently.

Although non-overcommit prevents running out of VM but when VM is full then system can either livelock or start arbitrary process killing so non-overcommit becomes useless. How others solved this? They reserve some VM for root so he can act whatever he wants. Well written apps (I could mention e.g. apache) don't really care about system is in an OOM situation - they happily do their jobs just as before (proved in practice ;)

When root reserved VM is also used up, welcome OOM killer - however with the above two protection bar the chance for this is pretty around 0 if admin don't prefer running stuffs as root.

Note, these are optional for those who are willing to sacrifice a couple of percent system resources.

Well, I ported the 2.4 OOM killer to 2.2.19 and added reserved root VM
http://mlf.linux.rulez.org/mlf/ezaz/reserved_root_memory.html

It works like it kills the process chosen by OOM killer when the first app can't get its page in page fault. OOM killing in 2.4 works differently. Apps loop forever in __alloc_pages() until they can get a page or out_of_memory() decides its time to kill somebody. So whenever there is some VM tuning, out_of_memory() should be tuned. Of course it's usually missed. Futhermore it's also a heuristic, not an "exact" decision made by e.g. 2.2 kernels. So in short, IMHO it will never work. I asked explanation for a couple of times for this 2.4 behavior but nobody bothered. I think partly it's because of the aggressive caching. One easy solution could be to drop out_of_memory() completely, put back oom_kill() to page fault from kswapd() and make tunable the number of looping in __alloc_pages().

Define to the user the "enough memory" when he wants to open different kind of documents, run scientific applications with different size data sets or whatever. What would you expect from your computer either "Hey, your resources is not enough for this task" or just crash the application?

The "buy more disk, buy more RAM!" kind of answers were one of the reasons Linux user base growth so big as it is today escaping from the old advisers ...

2.4 is killer if expertise is given [not to install Linux but to carefully setup the box for its job] but it fails otherwise because of its OOM handling.

Szaka

You people all have nice ideas. I guess we may want to preserve them for the future and don't let them get lost in a mailing list archive ;)

If you have the time, please create a page for these things in the Linux-MM wiki:

	http://linux-mm.org/wiki/

cheers,

Rik
--
IA64: a worthy successor to the i860.

		http://www.surriel.com/

http://www.conectiva.com/ http://distro.conectiva.com/

Followup to:  <Pine.LNX.4.33L.0108102347050.3530-100000@imladris.rielhome.conectiva>
By author:    Rik van Riel <[email protected]>
In newsgroup: linux.dev.kernel

Basically because there is no such thing?

-hpa

<[email protected]> at work, <[email protected]> in private!
"Unix gives you enough rope to shoot yourself in the foot."
http://www.zytor.com/~hpa/puzzle.txt	<[email protected]>

Actually the killer itself isn't the problem.

It's deciding when to let it kick in.

Rik
--
IA64: a worthy successor to i860.

http://www.surriel.com/ http://distro.conectiva.com/

Send all your spam to [email protected] (spam digging piggy)

I was under the presumption from what David and others have said that the OOM sometimes works the way it was meant to and kills the offending program, or it put the box into this super sluggish state for a very long time regardless
if it was early or late. If it only happens when it's late then what was
David talking about?

Perhaps a tunable load value w/ kswapd? If you're trying to accomplish more than N iterations of kswapd's particular function...take your pick, then make the OOM killer more trigger happy, perhaps on a sliding scale. At least then -something- will get killed the harder we try to get pages. As it is now, it's very likely the kernel get's stuck on itself for hours on end...sometimes never recovering. I suspect the only reason why I recovered it was because I happened to have about 8 ssh sessions to other machines that I was able to kill them on.

David

David, safemode,

your patches are appreciated. OOM is usually quite a rare condition so I won't be participating in any handwaving discussions.

I'm willing to discuss tested patches any time, however.

regards,

Rik
--
IA64: a worthy successor to i860.

http://www.surriel.com/ http://distro.conectiva.com/

Send all your spam to [email protected] (spam digging piggy)

Well... yeah...

-hpa

And also because

- people mix OOM and thrashing handling up - when they are logically
seperated questions.

- The 2.4 VM goes completely gaga under high load. Its beautiful under
light loads, there nobody can touch it, but when you actually really need it - splat.

So people either need to get an OOM when they are not but in fact might thrash, or the box thrashes so hard it makes insufficient progress to actually get out of memory.

OOM is also very hard to get right without reservations tracking in kernel for the journalling file systems and other similar stuff. To an extent thrash handling also wants RSS limits.

I understand this, I use reiserfs and because of this there is an increased baseline for "no more memory". This leads to your last paragraph below. The problem is that the OOM handler hasn't been made aware of this.

Don't we already measure VM pressure? It may be a hack but we could adjust the triggerpoint of the OOM handler on a sliding scale proportionate to the VM pressure, could we not? I think that is the most simple hack until we have a decent resource baseline in place..each subsystem would keep a tally of how much required memory it had/wanted and that way, the OOM handle would be much more intelligent about when to fire.

So this posits two questions.

- How hard/how much time.. is a sliding pressure scale hack to implement

- How hard/how much time.. to implement resource baselines (or similar concept)

David

Hi!

Maybe creating userland program that
*) allocates few megs
*) while 1 sleep 1m, gettimeofday. If more tha two minutes elapsed,
tell OOM handler to kick in.

Or maybe kernel could have some "VM watchdog", which would trigger OOM if it is not polled once a minute...
Pavel

Philips Velo 1: 1"x4"x8", 300gram, 60, 12MB, 40bogomips, linux, mutt,
details at http://atrey.karlin.mff.cuni.cz/~pavel/velo/index.html.

...

On my compute-server in the basement this is completely unacceptable because it *may* just be working hard on something big. The excessive swapping may just be 10-30 minutes where some app is using way more memory than the box has RAM, in this case it's no problem at all, and all your suggestion would give me is randomly dying compute jobs.

On my desktop this is unacceptable as well. You want the system to be frozen for more than two minutes before doing anything about it ?

The problem with using such vague heuristics against fixed (arbitrary) limits is that the effect will almost always be completely unacceptable. Either your arbitrary limit is way too high, or way too low.

I can't tell you how to do it - but I think your suggestion is an excellent way
to *not* do it :)

Didn't everyone pretty much agree that if we could turn off overcommit
completely and reliably, that would be the preferred solution ? Simply sig11
the app that's unlucky enough to want more memory than there's in the system (or, horror, have malloc() fail)

Now, I don't remember the entire thread, but IIRC it was difficult to kill overcommit completely.

................................................................
:   [email protected]   : And I see the elder races,         :
:.........................: putrid forms of man                :
:   Jakob Østergaard      : See him rise and claim the earth,  :
:        OZ9ABN           : his downfall is at hand.           :
:.........................:............{Konkhra}...............:

I think it is an excellent way to do it. Nobody said you have to run the program and nobody forces you to use a particular program with a particular policy. It puts the OOM policy in userland where -you- decide when and how things happen.

David

The kernel allocates memory within itself. We will still reach OOM conditions. It can't be avoided.

David

Sure - what I was trying to say was, that I don't think the solution will work very well.

...

Good point.

................................................................
:   [email protected]   : And I see the elder races,         :
:.........................: putrid forms of man                :
:   Jakob Østergaard      : See him rise and claim the earth,  :
:        OZ9ABN           : his downfall is at hand.           :
:.........................:............{Konkhra}...............:

Nope, not at all. The kernel acquires and releases memory as it goes. My statement is to the fact that we can reach a point where we have exhausted all the memory resources by the time we start a particular code path but in order to complete that code path we need more memory. I.e. journaled filesystems. We have reached 0 memory but we need to start down a code path to update data on the disk. That means we may have to allocate memory to read and update the journal as we start writing regular file data to disk.

This only occurs when we ride right on the very edge of OOM. This will also be fixed when we implement all the desired bean counting.

David