1gb RAM + 1gb SWAP + make -j bzImage = OOM

On 2.4.17, I can't make -j bzImage without OOM kicking in. Relatively light .config here - bzImage compiles to less than 1mb.

Seems with 1 gb of RAM and swap, the box should be able to handle this (box is dual P3 600 btw).

Is this unreasonable? How much RAM should it take to accomplish this???

-Phil Oester

You should give a bit more info on that, especially vmstat and the like. I cannot reproduce this. Neither on 1GB/256MB nor on 2GB/256MB RAM/SWAP. (P3-1GHz, dual SMP, 2.4.17)

Regards,
Stephan

I have absilutely no trouble reproducing on an 800MHz Athlon with 256MB RAM/256MB swap on 2.4.17

The one catch is that -j is specified without a number.

from man make:
-j jobs
Specifies the number of jobs (commands) to run
simultaneously. If there is more than one -j
option, the last one is effective.
**If the -j option is given without an argument, make will not limit the number of jobs that can run simultaneously.**

(emphasis mine)

Hence, unlimited number of jobs, theoreticaly unlimited amount of memory usage.
The last number of processes I saw in top before the system was basically dead and I just hit A-SYSRQ-S and A-SYSRQ-B was 416, and all the top processes were make or cc

Somehow I doubt this is a kernel issue and is instead a make and user issue. A make issue because it's probably poor design to have an option that's specified with a number be normaly harmless and useful, be potentialy lethal when the number is left off, so if you forget the number, your system is dead. A user issue because it seems the user is using the option without fully comprehending the consequences.

[snip superfluous description of what 'make -j' implies]

eh? Trust me - i understand the implications of make -j. It's not an unreasonable test, especially on a machine with 1gb ram/swap. For reference, read Rik's email regarding his reverse VM patch:

http://marc.theaimsgroup.com/?l=linux-kernel&m=101007711817127&w=2

Might be enlightening

-Phil

No troubles to reproduce this here, on sparc64 !GM ran/1GB swap, and on dualathlon 768MB RAM 1.5GB swap, and on athlon 1GBRAM/1GBSWAP

But this is not a kernel issue, it is simply that
too many gcc processes are runned at the same time because the source files are too many.

unreasonable test, especially on a machine with 1gb ram/swap. For reference, read Rik's email regarding his reverse VM patch:>

I guess this testcase is somewhat driving in the direction of Martins test with some setis running, meaning it has a lot of standard processes that need files and try to work out something. Can you try Martins patch at your side, redo the -j story and give us a result? I attached it for an easy go :-)

Thanks,
Stephan

@@ -394,9 +394,9 @@
 		if (PageDirty(page) && is_page_cache_freeable(page) && page->mapping) {
 			/*
 			 * It is not critical here to write it only if
-			 * the page is unmapped beause any direct writer
+			 * the page is unmapped because any direct writer
 			 * like O_DIRECT would set the PG_dirty bitflag
-			 * on the phisical page after having successfully
+			 * on the physical page after having successfully
 			 * pinned it and after the I/O to the page is finished,
 			 * so the direct writes to the page cannot get lost.
 			 */

@@ -480,11 +480,14 @@
 
 			/*
 			 * Alert! We've found too many mapped pages on the
-			 * inactive list, so we start swapping out now!
+			 * inactive list.
+			 * Move referenced pages to the active list.
 			 */
-			spin_unlock(&pagemap_lru_lock);
-			swap_out(priority, gfp_mask, classzone);
-			return nr_pages;
+			if (PageReferenced(page)) {
+				del_page_from_inactive_list(page);
+				add_page_to_active_list(page);
+			}
+			continue;
 		}
 
 		/*

@@ -521,6 +524,9 @@
 	}
 	spin_unlock(&pagemap_lru_lock);
 
+	if (max_mapped <= 0 && nr_pages > 0)
+		swap_out(priority, gfp_mask, classzone);
+
 	return nr_pages;
 }
 

The simple question is: is the RAM sufficient at all to spawn such a lot of cc processes? In my setup I get around 1000 concurrently working during -j. This sounds like a real problem for 256/256, or not?

Regards,
Stephan

Matter of scale, did you try a full kernel build with make -j bzImage using whatever your normal config is?
I still believe this is an innappropriate test, sure if you have tons of RAM and swap it may eventualy complete (though if the swapfile is very active and this is tried on a 2.4 kernel, the amount of CPU time the compile will actually get will likely be unpredictable at best.) This is an option that does nothing less than flood the system with hundreds or thousands of processes that on any large compile will simply not allow the system to survive intact without killing all the processes. Does the OOM killer even work correctly yet? Or does it still try to kill init at times? And for that matter, what order does it kill in?

Yes. It sounds like he Rick slowed down fork enough the system didn't fall over. There may be some other policy changes as well. But my hunch is that it is a fork speed thing. If all that happens is that the system hits OOM when subjected to an unreasonable load I don't see this as a problem.

The truly interesting question is what happens when you and more swap. With sufficient swap will it work?

Eric

Yes, of course, and it works at my side. Worked with 1GB RAM/256MB swap, works now with 2GB RAM/256MB swap on stock 2.4.17.

I never saw it not completing on my box with 2.4.17, regardless of what I do in the mean time (writing mails or the like). Of course system performance drops somehow down when load reaches about 150, but I think this can be expected ;-)

Regards,
Stephan

I've rerun this test a number of times, and cannot reliably reproduce the OOM - though it still does OOM occasionally. It never OOM's right after a bootup - usually the greatest chance of OOM is after 2 or 3 consecutive runs without a reboot. Once it even froze the box and required a powercycle.

I'm surprised you cannot OOM with 1gb RAM/256MB swap, as sometimes I'm over 900MB in swap - did you try consecutive runs, or just once and then reboot between each run?

On a side note, there seems to be some debate as to whether this is a valid test. The detractors primarily claim that 'make -j' just overloads the machine with too many processes and therefore is setting it up to fail. My position has always been that the kernel _should_not_OOM_ under this test due to the ~2gb of ~RAM being thrown at it. It may die for any number of other reasons, but OOM shouldn't be one of them. In other words, either the OOM killer may be too aggressive here, or the kernel isn't reclaiming inactive RAM under heavy load.

Haven't yet tried Martin's patch - though since I can't reliably produce the OOM, testing it wouldn't help much.

-Phil Oester

-----Original Message-----
From: Stephan von Krawczynski [mailto:[email protected]]
Sent: Saturday, January 05, 2002 7:17 AM
To: Phil Oester
Cc: [email protected]; [email protected]
Subject: Re: 1gb RAM + 1gb SWAP + make -j bzImage = OOM

I guess this testcase is somewhat driving in the direction of Martins test with
some setis running, meaning it has a lot of standard processes that need files
and try to work out something. Can you try Martins patch at your side, redo the
-j story and give us a result? I attached it for an easy go :-)

Thanks,
Stephan

I tried just about everything I could think of and it never went in OOM. Even the first test I did were with several days uptime - meaning far away from "cleaning" reboot. I hate reboot :-)

Well, take the other side: if you do not manage to OOM afterwards, even at the tenth consecutive try, there is probably something about the patch ...

Regards,
Stephan

The vmscan patch doesn't seem to help in the 'make -j' testcase.

Here's time of a couple runs:

2.4.17 vanilla

real    32m2.097s
user    9m51.800s
sys     3m47.700s

real    19m45.696s
user    9m55.820s
sys     2m32.170s

2.4.17 + vmscan patch

gave up waiting after 2 hours...never finished.

Unfortunately, box was not responsive enough to gather any useful information. Perhaps not swapping enough???

-Phil

-----Original Message-----
From: Stephan von Krawczynski [mailto:[email protected]]
Sent: Monday, January 07, 2002 6:24 AM
To: Phil Oester
Cc: [email protected]; [email protected]
Subject: Re: 1gb RAM + 1gb SWAP + make -j bzImage = OOM

then

I tried just about everything I could think of and it never went in OOM. Even
the first test I did were with several days uptime - meaning far away from
"cleaning" reboot. I hate reboot :-)

produce

Well, take the other side: if you do not manage to OOM afterwards, even at the
tenth consecutive try, there is probably something about the patch ...

Regards,
Stephan