VM and swap - it's really unusable

Hello all,

Again, I did a rsync-operation as described in "[2.4.17rc1] Swapping" MID <[email protected]>.

This time, the kernel had a swappartition which was about 200MB. As the swap-partition was fully used, the kernel killed all processes of knode. Nearly 50% of RAM had been used for buffers at this moment. Why is there so much memory used for buffers?

I know I repeat it, but please:

Fix the VM-management in kernel 2.4.x. It's unusable. Believe me! As comparison: kernel 2.2.19 didn't need nearly any swap for the same operation!

Please consider that I'm using 512 MB of RAM. This should, or better: must be enough to do the rsync-operation nearly without any swapping - kernel 2.2.19 does it!

The performance of kernel 2.4.18pre1 is very poor, which is no surprise, because the machine swaps nearly nonstop.

Regards,
Andreas Hartmann

If you feel adventurous you can try my rmap based
VM, the latest version is on:

	http://surriel.com/patches/2.4/2.4.17-rmap-8

This VM should behave a bit better (it does on my machines), but isn't yet bug-free enough to be used on production machines. Also, the changes it introduces are, IMHO, too big for a stable kernel series ;)

regards,

Rik

DMCA, SSSCA, W3C?  Who cares?  http://thefreeworld.net/

http://www.surriel.com/		http://distro.conectiva.com/

Basically, I'm using NFS and reiserfs. But I didn't use any file on NFS since the last reboot - and the NFS-shares haven't been mounted.

There are 2 IDE-Harddisks in this machine:
hda: WDC WD205AA, ATA DISK drive (40079088 sectors (20520 MB) w/2048KiB

				  cache, CHS=2494/255/63, UDMA(66))
hdb: WDC WD450AA-00BAA0, ATA DISK drive (87930864 sectors (45021 MB)
					w/2048KiB Cache,
					CHS=5473/255/63, UDMA(66))

On hda, I have got 7 partitions (plus one little "boot"-partition, which isn't mounted and a 200MB swap partition). On hdb, I have got 12 partitions and one more, meanwhile 1GB swap partition.
All partitions are formated with reiserfs.

Regards,
Andreas Hartmann

Does the 2.4.9 Red Hat kernel (if yoiu are using RH) or 2.4.12-ac8 show the same problem ?

Some other examples:
I just did a
cp -Rd linux-2.4.16 linux-2.4.17
(with object-files). Before starting this action, I had about 120 MB of free RAM. During copying - I did nothing else meanwhile, there was 2MB swap used - and 12 MB of RAM were free. The biggest part of memory was used for caching - what is ok. After copying, only 10 MB of memory have been given free again. There have been 490MB of RAM used now (nearly most for caching).

Outgoing from this situation, I started another little cp-action:
cp -Rd linux-2.4.18pre1 linux-2.4.test
(again including object files).
Result: the swap usage stayed nearly constant, neverthless there have been 6 accesses to swap.

Now, I deleted the linux-2.4.test-directory with
rm -R linux-2.4.test
This action was very fast (approximately 1s).

Afterwards, a big part of the cache memory has been given free (about 100MB). Now, 122MB of RAM have been free again.

Next example (running after the last):
SuSE run-crons have been running. This means:
-> updatedb
-> sort
-> frcode
-> find
-> mandb

47MB swap used, 2/3 of memory is used for buffers (Don't forget: I've got 512MB of RAM) and about 30MB of RAM are free.

My observation:
Why does the kernel swap to get free memory for caching / buffering? I can't see any sense in this action. Wouldn't it be better to shrink the cashing / buffering-RAM to the amount of memory, which is obviously free?

Swapping should be principally used, if the RAM ends for real memory (memory, which is used for running applications). First of all, the memory-usage of cache and buffers should be reduced before starting to swap IMHO.

Or would it be possible, to implement more than one swapping strategy, which could be configured during make menuconfig? This would give the user the chance to find the best swapping strategy for his purpose.

Regards,
Andreas Hartmann

please try to reproduce on 2.4.17rc2aa2, thanks.

	ftp://ftp.us.kernel.org/pub/linux/kernel/people/andrea/kernels/v2.4/2.4.17rc2aa2.bz2

Andrea

Unfortunately, I lost the response that basically said "2.4 looks stable to me", but let me count the ways in which I agree with Andreas' sentiment:

A) VM has major issues
1) about a dozen recent OOPS reports in VM code
2) VM falls down on large-memory machines with a high inode count (slocate/updatedb, i/dcache)
3) Memory allocation failures and OOM triggers even though caches remain full.
4) Other bugs fixed in -aa and others
B) Live- and dead-locks that I'm seeing on all 2.4 production machines > 2.4.9, possibly related to A. But how will I ever find out?
C) IO-APIC code that requires noapic on any and all SMP machines that I've ever run on.

I don't have anything against anyone here -- I think everyone is doing a fine job. It's an issue of acceptance of the problem and focus. These issues are all showstoppers for me, and while I don't represent the 90% of the Linux market that is UP desktops, IMHO future work on the kernel will be degraded by basic functionality that continues to cause problems.

I think seeing some of Andrea's and Andrew's et al patches actually *happen* would be a good thing, since 2.4 kernels are decidedly not ready for production here. I am forced to apply 26 distinct patch sets to my kernels, and I am NOT the right person to make these judgements. Which is why I was interested in an LKML summary source, though I haven't yet had a chance to catch up on that thread of comment.

Having a glitch in the radeon driver is one thing; having persistent, fatal, and reproducable failures in universal kernel code is entirely another.

Ken.
[email protected]

On Fri, Dec 28, 2001 at 09:16:38PM +0100, Andreas Hartmann wrote:

| Hello all,
| 
| Again, I did a rsync-operation as described in
| "[2.4.17rc1] Swapping" MID <[email protected]>.
| 
| This time, the kernel had a swappartition which was about 200MB. As the 
| swap-partition was fully used, the kernel killed all processes of knode.
| Nearly 50% of RAM had been used for buffers at this moment. Why is there 
| so much memory used for buffers?
| 
| I know I repeat it, but please:
| 
| 	Fix the VM-management in kernel 2.4.x. It's unusable. Believe
| 	me! As comparison: kernel 2.2.19 didn't need nearly any swap for
| 	the same operation!
| 
| Please consider that I'm using 512 MB of RAM. This should, or better: 
| must be enough to do the rsync-operation nearly without any swapping - 
| kernel 2.2.19 does it!
| 
| The performance of kernel 2.4.18pre1 is very poor, which is no surprise, 
| because the machine swaps nearly nonstop.
| 
| 
| Regards,
| Andreas Hartmann
| 
| -
| To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
| the body of a message to [email protected]
| More majordomo info at  http://vger.kernel.org/majordomo-info.html
| Please read the FAQ at  http://www.tux.org/lkml/

I've spent ages trying to fix these bugs in the -ac kernel, but they got all backed out in search of better performance.

Right now I'm developing a VM again, but I have no interest at all in fixing the livelocks in the main kernel, they'll just get removed again after a while.

If you want to test my VM stuff, you can get patches from
http://surriel.com/patches/ or direct access at the bitkeeper tree on http://linuxvm.bkbits.net/

cheers,

Rik

Shortwave goes a long way:  irc.starchat.net  #swl

http://www.surriel.com/		http://distro.conectiva.com/

Ben LaHaise's fix for page_cache_release() is absolutely required.

Does this happen with the latest -aa patch? If so, please send a full system description and report.

Dunno about this one. Have you prepared a description?

-

stable

On all boxes I run currently (all 1GB or below RAM), I cannot find _major_ issues.

Must be beyond the GB range.

I have not had one up to now in everyday life with 2.4.17

Hm, well I would expect Andrea to do tuning and fixing as experience evolves...

Me = none up to now I could track down to a kernel issue. The single one I had was with a distro kernel around 2.4.10 and flaky hardware.

I am currently running 5 Asus CUV4X-D based SMP boxes all with apic _on_, amongst which are squids, sql servers, workstation type setups (2 my very own).

doing a

These

90%

kernel

Have you run _yourself_ into a problem with 2.4.17?
I mean it is not perfect of course, but it is far better than you make it look.
I could hand the brown bag to all versions below about 2.4.15 pretty easy, but since 2.4.16 it has really become hard to shoot it down for me. Ok, I use only pretty selected hardware, but there are reasons I do, and they are not related to the kernel in first place.

Regards,
Stephan

Actually, I posted about C) many moons ago, and had some chats with Manfred Spraul and Alan. It's a tough one to crack, and I have my own workaround patch (below) that I've been using for a while now. My posts are in the archives, but I can send a summary by request.

I haven't succeeded my bag check in putting -aa in production, which is where I'm able to reproduce these problems. Part of the problem is me, in that I can't easily test with -aa. And part of the problem is chicken vs egg -- can't test unless it's in mainline, don't want to put questionable stuff in a release kernel, even a -pre... But I do think the -aa stuff is worth breaking out into Marcelo-digestable chunks as soon as Andrea can.

The machines that are OOPSing are in production and right now don't have serial consoles available... that will change in a month or so, but right now I can't decode OOPSes without hand-copying. I might get that desparate unless the problem goes away with 2.4.18 (with -aa merged, hopefully. :)

Thanks much,

Ken.
[email protected]


Applies to any recent 2.4.  Changing indent sucks.

--- linux/arch/i386/kernel/io_apic.c.orig	Tue Nov 13 17:28:41 2001
+++ linux/arch/i386/kernel/io_apic.c	Tue Dec 18 15:10:45 2001
@@ -172,6 +172,7 @@
 int pirq_entries [MAX_PIRQS];
 int pirqs_enabled;
 int skip_ioapic_setup;
+int pintimer_setup;
 
 static int __init ioapic_setup(char *str)
 {
@@ -179,7 +180,14 @@
 	return 1;
 }
 
+static int __init do_pintimer_setup(char *str)
+{
+	pintimer_setup = 1;
+	return 1;
+}
+
 __setup("noapic", ioapic_setup);
+__setup("pintimer", do_pintimer_setup);
 
 static int __init ioapic_pirq_setup(char *str)
 {
@@ -1524,27 +1532,31 @@
 		printk(KERN_ERR "..MP-BIOS bug: 8254 timer not connected to IO-APIC\n");
 	}
 
-	printk(KERN_INFO "...trying to set up timer (IRQ0) through the 8259A ... ");
-	if (pin2 != -1) {
-		printk("\n..... (found pin %d) ...", pin2);
-		/*
-		 * legacy devices should be connected to IO APIC #0
-		 */
-		setup_ExtINT_IRQ0_pin(pin2, vector);
-		if (timer_irq_works()) {
-			printk("works.\n");
-			if (nmi_watchdog == NMI_IO_APIC) {
-				setup_nmi();
-				check_nmi_watchdog();
+	if ( pintimer_setup )
+		printk(KERN_INFO "...skipping 8259A init for IRQ0\n");
+	else {
+		printk(KERN_INFO "...trying to set up timer (IRQ0) through the 8259A ... ");
+		if (pin2 != -1) {
+			printk("\n..... (found pin %d) ...", pin2);
+			/*
+			 * legacy devices should be connected to IO APIC #0
+			 */
+			setup_ExtINT_IRQ0_pin(pin2, vector);
+			if (timer_irq_works()) {
+				printk("works.\n");
+				if (nmi_watchdog == NMI_IO_APIC) {
+					setup_nmi();
+					check_nmi_watchdog();
+				}
+				return;
 			}
-			return;
+			/*
+			 * Cleanup, just in case ...
+			 */
+			clear_IO_APIC_pin(0, pin2);
 		}
-		/*
-		 * Cleanup, just in case ...
-		 */
-		clear_IO_APIC_pin(0, pin2);
+		printk(" failed.\n");
 	}
-	printk(" failed.\n");
 
 	if (nmi_watchdog) {
 		printk(KERN_WARNING "timer doesnt work through the IO-APIC - disabling NMI Watchdog!\n");

On Thu, Jan 03, 2002 at 01:54:14PM -0800, Andrew Morton wrote:

| Ken Brownfield wrote:
| > 
| > Unfortunately, I lost the response that basically said "2.4 looks stable
| > to me", but let me count the ways in which I agree with Andreas'
| > sentiment:
| > 
| >         A) VM has major issues
| >                 1) about a dozen recent OOPS reports in VM code
| 
| Ben LaHaise's fix for page_cache_release() is absolutely required.
| 
| >                 2) VM falls down on large-memory machines with a
| >                    high inode count (slocate/updatedb, i/dcache)
| >                 3) Memory allocation failures and OOM triggers
| >                    even though caches remain full.
| >                 4) Other bugs fixed in -aa and others
| >         B) Live- and dead-locks that I'm seeing on all 2.4 production
| >            machines > 2.4.9, possibly related to A.  But how will I
| >            ever find out?
| 
| Does this happen with the latest -aa patch?  If so, please send
| a full system description and report.
| 
| >         C) IO-APIC code that requires noapic on any and all SMP
| >            machines that I've ever run on.
| 
| Dunno about this one.  Have you prepared a description?
|  
| 
| -

On Fri, Jan 04, 2002 at 01:19:28AM +0100, Stephan von Krawczynski wrote:

| > A) VM has major issues                                              
|                                                                       
| On all boxes I run currently (all 1GB or below RAM), I cannot find    
| _major_ issues.                                                       

Yeah, I'm seeing it primarily with 1-4GB, though I have very few <1GB machines in production.

| > 2) VM falls down on large-memory machines with a                    
| >    high inode count (slocate/updatedb, i/dcache)                    
|                                                                       
| Must be beyond the GB range.                                          

The critical part is the high inode count -- memory amount increases the severity rather than triggering the problem.

| > 3) Memory allocation failures and OOM triggers                      
| >    even though caches remain full.                                  
|                                                                       
| I have not had one up to now in everyday life with 2.4.17             

I'm seeing this in malloc()-heavy apps, but fairly sporadic unless I create a test case. On desktops, most of these issues disappear, but I do think the mindset behind the kernel needs to at least partially break free of the grip of UP desktops, at least to the point of fixing issues like I'm mentioning.

Not critical for me; but high-profile on lkml.

[...]

| > C) IO-APIC code that requires noapic on any and all SMP             
| >   machines that I've ever run on.                                   
|                                                                       
| I am currently running 5 Asus CUV4X-D based SMP boxes all with apic   
| _on_, amongst  which are squids, sql servers, workstation type setups 
| (2 my very own).                                                      

Do they have *sustained* heavy hit/IRQ/IO load? For example, sending 25Mbit and >1,000 connections/s of sustained small images traffic through khttpd will kill 2.4 (slow loss of timer and eventual total freeze) in a couple of hours. Trivially reproducable for me on SMP with any amount of memory. On HP, Tyan, Intel, Asus... etc.

| Have you run _yourself_ into a problem with 2.4.17?                   
| I mean it is not perfect of course, but it is far better than you make
| it look.                                                              

2.4.17 (and -pre/-rc) is my yardstick, actually. With the exception of -aa, I stay very close to the bleeding edge.

Please don't misunderstand -- I don't think any 2.4 kernel sucks (with the exception of the two or three DONTUSE kernels. :) In fact, I have zero complaints other than the ones I've listed. I was ecstatic when 2.2 came out, and 2.4 is just as impressive.

It's not that the kernel is bad, it's that there are specific things that shouldn't be forgotten because of a "the kernel is good" evaluation. Especially those that make Linux regularly unstable in common production environments.

| I could hand the brown bag to all versions below about 2.4.15  pretty 
| easy, but since 2.4.16 it has really become hard to shoot it down for 
| me. Ok, I use only pretty selected hardware, but there are reasons I  
| do, and they are not related to the kernel in first place.            

I use pretty selected hardware as well -- scaling hundreds of servers for varied uses really depends on having someone track and select hardware, and using it homogenously. Of course, of all of the selected hardware I've used over the last two years since 2.4.0-test1, C) has persisted on all configurations, but the others are more recent but equally omnipresent.

Like I said, I suspect that most people with machines in lower-load environments don't have these issues, but "number of people effected" is only one metric to judge the importance of an issue.

Of course, I'm not biased or anything. ;-)

Thanks for the input,

Ken.
[email protected]


|                                                                       
| Regards,                                                              
| Stephan                                                               
|                                                                       

On Thu, Jan 03, 2002 at 11:26:01PM -0600, you [Ken Brownfield] claimed:

I'm seeing this on 2GB IA64 (2.4.16-17). I posted a _very_ simple test case to lkml a while a go. It didn't happen on 256MB x86.

I plan to try -aa shortly, now that I got patches to make it compile on IA64.

-- v --

[email protected]

Ok. It would be really nice to know if the -aa patches do any good at your configs. Andrea has possibly done something on the issue. But let me take this chance to state an open word: last time Andrea talked about his personal hardware I couldn't really believe it - because it was so ridiculously small. I wonder if anyone at SuSE management _does_ actually read this list and think about how someone can do a good job without good equipment. If you really want to do something groundbreaking about highmem you have to have a _box_. A box _somewhere_ in the world or a patch for highmem-in-lowmem is not really the same thing. Even Schumacher wouldn't have won formula one by sitting inside a Fiat Uno with a patched speedometer.

You are right.

Hm, I have about 24GB of NFS traffic every day, which may be too less. What exactly are you seeing in this case (logfiles etc.)?

Hopefully nobody does this here, I don't.

The number of people is not really interesting for me, as the boxes get bigger every day it is only a matter of time to see more people with lots of GB (as an example).

How could you ? ;-))

Regards,
Stephan

Ok, I am going to buy more mem right now to see what you see.

Regards,
Stephan

Unfortunately you are right.

Question is: which nature is your application / load of the system? You wrote something about database server. How much rows alltogether? What's the size of the table(s)? How many concurrent accesses do you have? Do you do "easy" searches where all of the conditions are located in the index? How big is your index? How big is the throughput of your database? Do you have your tables on raw partitions (without caching; as you can do it with UDB)?

You mentioned squid, too. I'm running squid here on a AMD K6 2 400, 256 MB RAM. It's mostly (sometimes plus my wife) for my own. No more users. In this situation, I can't see any problem, too. Why? There is no load, no throughput, ... .

How big are the partitions you are mounting at once? In my case, all the partitions together have about 70GB (all reiserfs).

I want to know it, because I think the problem depends on how much different HD-memory is accessed. If you have applications, which doesn't access to much memory, you can't view the problems. If you access more than 1G (and you do not just copy, but rsync e.g.) and you have only 512MB of RAM, the machine swaps a lot with most actual 2.4.-kernels (patches).

Another question:
Are there any tools to meassure the datathroughput a application causes? Interesting would be the sum at the end of the process, the maximum and average throughput (in- and output seperated) and the same for swapactivity. It could probably help to find optimization potential. At least it would give the chance to directly compare the demand of different applications.

Regards,
Andreas Hartmann

On Fri, 04 Jan 2002 21:15:42 +0100
Andreas Hartmann <[email protected]> wrote:

[I will answer not all of your questions, as this is a matter of business, too]

Generally we do not drive the boxes up to the edge. Our philosophy is to throw money at the problem, before it actually arises. Yes, I can see the future ... ;-)

No.

about 130 GB, all reiserfs.

I guess you should tilt that theory.
Have you already tried to throw a big SPARC at the problem?

Can you provide a simple and reproducible test case (e.g. some demo source), where things break? I am very willing to test it here.

Regards,
Stephan

On Fri, Jan 04, 2002 at 02:03:21PM +0100, Stephan von Krawczynski wrote:
[...]
| Ok. It would be really nice to know if the -aa patches do any good at your

I'd love to, but unfortunately my problems reproduce only in production, and -- nothing against Andrea -- I'm hesitant to deploy -aa live, since it hasn't received the widespread use that mainline has. I may be forced to soon if the VM fixes don't get merged.

[...]

| > Do they have *sustained* heavy hit/IRQ/IO load?  For example, sending
| > 25Mbit and >1,000 connections/s of sustained small images traffic
| > through khttpd will kill 2.4 (slow loss of timer and eventual total
| > freeze) in a couple of hours.  Trivially reproducable for me on SMP with
| > any amount of memory.  On HP, Tyan, Intel, Asus... etc.
| 
| Hm, I have about 24GB of NFS traffic every day, which may be too less. What
| exactly are you seeing in this case (logfiles etc.)?

Well, the nature of the problem is that the timer "slows" and stops, causing the machine to get more and more sluggish until it falls of the net and stops dead.

I suspect that high IRQ rates cause the issue -- large sequential transfers are not necessarily culprits due the lowish overhead.

[...]

| > It's not that the kernel is bad, it's that there are specific things
| > that shouldn't be forgotten because of a "the kernel is good"
| > evaluation.
| 
| Hopefully nobody does this here, I don't.

I don't think it's intentional, and I realize that VM changes are hard to swallow in a stable kernel release. I just hope that the severity and fairly wide negative effect is enough to make people more comfortable with accepting VM fixes that may be somewhat invasive.

Thanks,

Ken.
[email protected]

Stephan von Krawczynski wrote:

[...]

It's easy - take a grown inn-newsserver-partition with reiserfs (*) (a lot of small files and a lot of directories), about 1,3 GB or more, and do a complete rsync to this partition to transport it somewhere else. But you have to do it with a existing target, no empty target, so that rsync must scan the whole target partition, too.

I don't like special test-programs. They seldom show up the reality. What we need is a kernel that behaves fine in reality - not in testcases. And before starting the test, take care, that most of ram is already used for cache or buffers or applications.

I did this test with several VM-patches and there are huge differences in swap consumption between them: 319MB with 2.4.17rc2 and 59MB with 2.4.17 oom-patch (max). It's more than a little difference :-).

Regards,
Andreas Hartmann

(*) If I had DSL, I would send it to you (as tar.gz) - but with modem, it's a bit too much :-)!
But your squid cache should be fine, too. It has a similar structure: a lot of small files and a lot of subdirectories. But I think, that your squid cache size isn't as high as my inn-partition.

"We" (Auctionwatch.com) are experiencing problems that appear to be related to VM, I realize that this question was not directed at me:

Mysql running a dual 650 PIII, 2 gig ram. Rows? Dunno, but several million tables (about 85 gig of tables averaging 45-50k IIRC).

We will have 2-400+ tables open at once.

I don't know much about the specific design, other than I've been told it's non-optimal.

One 250G logical volume, a couple smaller ones (3 gig, 30 gig).

Share and Enjoy. 

OK, here's some pseduo-code for a real-world test case. I haven't had a chance to code it up, but I'm guessing I know what it's going to do. I'd *love* to be proved wrong :).

# build and boot a kernel with "Magic SysRq" turned on
# echo  1 > /proc/sys/kernel/sysrq
# fire up "nice --19 top" as "root"
# read "MemTotal" from /proc/meminfo

# now start the next two jobs concurrently

# write a disk file with "MemTotal" data or more in it

# perform a 2D in-place FFT of total size at least "MemTotal/2" but less
# than "MemTotal"

Watch the "top" window like a hawk. "Cached" will grow because of the disk write and "free" will drop because the page cache is growing and the 2D FFT is using *its* memory. Eventually the two will start competing for the last bits of free memory. "kswapd" and "kupdated" will start working furiously, bringing the system CPU utilization to 99+ percent. At this point the system will appear highly unresponsive.

Even with the "nice --19" setting, "top" is going to have a hard time keeping its five-second screen updates going. You will quite possibly end up going to the console and doing alt-sysrq-m, which dumps the memory status on the console and into /var/log/messages. Then if you do alt-sysrq-i, which kills everything but "init", you should be able to log on again.

I'm going to try this on my 512 MB machine just to see what happens, but I'd like to see what someone with a larger machine, say 4 GB, gets when they do this. I think attempting to write a large file and do a 2D FFT concurrently is a perfectly reasonable thing to expect an image processing system to do in the real world. A "traditional" UNIX would do the I/O of the file write and the compute/memory processing of the FFT together with little or no problem. But because the 2.4 kernel insists on keeping all those buffers around, the 2D FFT is going to have difficulty, because it has to have its data in core.

What's worse is if the page cache gets so big that the FFT has to start swapping. For those who aren't familiar with 2D FFTs, they take two passes over the data. The first pass will be unit strides -- sequential addresses. But the second pass will be large strides -- a power of two. That second pass is going to be brutal if every page it hits has to be swapped in!

The solution is to limit page cache size to, say, 1/4 of "MemTotal", which I'm guessing will have a *negligible* impact on the performance of the file write. I used to work in an image processing lab, which is where I learned this little trick for bringing a VM to its knees, and which is probably where the designers of other UNIX systems learned that the memory used for buffering I/O needs to be limited :). There's probably a VAX or two out there still that shudders when it remembers what I did to it. :))

M. Edward Borasky

[email protected]
http://www.borasky-research.net

I am pretty impressed by Martins test case where merely all VM patches fail with the exception of his own :-) The thing is, this test is not of nature "very special" but more like "system driven to limit by normal processes". And this is the real interesting part about it.

What exactly do you mean with "high IRQ rate"? Can you show so numbers from /proc/interrupts and uptime for clarification?

Hm, I don't think real "big" patches are needed, Rik is according to Martins test no gain currently as rmap flops in this test, too.

The problem is: you should really use one of your problem machines for at least very simple testing. If you don't you possibly cannot expect your problem to be solved soon. We would need input from your side. If I were you, I'd start of with Martins patch. It is simple (very simple indeed), small and pinned to a single procedure. Martins test shows - under "normal" high load (not especially IRQ) - good result and no difference in standard load, I cannot see a risk for oops or deadlock.

Regards,
Stephan

And how exactly do the problems look like?

Regards,
Stephan

I would like to try it with the oom-patch, which needed less swap in my tests. It could be a good test to verify the results of the rsync-test.

Could you please tell me a programm, that does 2D FFT? I would like to do this test, too!

Regards,
Andreas Hartmann

Try http://www.fftw.org. This is a public domain (GPL I think) general purpose FFT library. If I get a chance I'll download it this weekend and figure out how to code a 2D FFT.

--
M. Edward Borasky

[email protected]
http://www.borasky-research.net

Never play leapfrog with a unicorn.

On Sat, Jan 05, 2002 at 04:08:33PM +0100, Stephan von Krawczynski wrote:

| I am pretty impressed by Martins test case where merely all VM patches fail
| with the exception of his own :-) The thing is, this test is not of nature
| "very special" but more like "system driven to limit by normal processes". And
| this is the real interesting part about it.

One problem is that I've never heard of it and don't know where to get it. ;)

| What exactly do you mean with "high IRQ rate"? Can you show so numbers from | /proc/interrupts and uptime for clarification?

I did, back in the archives. I don't have easy access to archives etc, right now, but I might repost since it's been a while.

| The problem is: you should really use one of your problem machines for at least
| very simple testing. If you don't you possibly cannot expect your problem to be
| solved soon. We would need input from your side. If I were you, I'd start of
| with Martins patch. It is simple (very simple indeed), small and pinned to a
| single procedure. Martins test shows - under "normal" high load (not especially
| IRQ) - good result and no difference in standard load, I cannot see a risk for
| oops or deadlock.

Well, reboots are the problem over possible oopses (or data corruption, even more fun.) But on your recommendation I'll give Martin's mod a try, given a URL. Does Martin's patch play well with -aa? How about Martin+10_vm in -pre2? ;-)

At any rate, right now there are three or four people with different VM patch sets, probably more. There is a certain amount of work this group can do in judging which concepts are cleaner or most suitable to 2.4.x. It would be cool to give rmap a try, but I don't want to maintain a 2.4.x kernel with speculative features that aren't intented for 2.4.x.

I can see using patches back-ported from 2.5, but I'm a firm believer that 2.4 should stay stable and that the benefit of 2.4 to admins is the control by the maintainer and stability -- not the VM of the month.

I can test, but it's slow going with so many patches. And many of the patches haven't been properly merged with any kernel (e.g., -aa 10_vm reverting previously applied 2.4 changes, etc.)

While I've reproduced the issues and explained them here in the past, it's difficult for me to iterate fast enough in an environment that easily reproduces tha problem. I'm iterating as fast as I can, but when I do iterate I'd prefer some support from the maintainers or other parts of the community that "Yes, this patch has a good chance of fixing the specific problems we've been seeing, give it a try." Right now that doesn't exist (with the exception of your recommendation of this Martin patch), and that's one reason I'm hesitant to iterate too much and effect a lot of people.

Thanks,

Ken.
[email protected]


| 
| Regards,
| Stephan
| 

Like dd if=/dev/zero of=/tmp/file bs=... count=... ?

I'm willing to try. What program can I use for FFT?

Can you describe FFT memory access pattern in more detail? I'd like to write a simple testcase with similar 'bad' pattern. --
vda

[Sent in off-LKML mail]

I read all your LKML mails since beginning of November, could find a lot about cpu, configs,tops etc but not a single "cat /proc/interrupts" together with uptime.

According to the ongoings of your mails you seem to try really a lot of things to make it work out. I recommend not to intermix the patches a lot. I would stay close to marcelo's tree and try _single_ small patches on top of that. If you mix them up (even only two of them) you won't be able to track down very well, what is really better or worse.

One thing I would like to ask here is this (as you are dealing with oracle stuff): why does oracle recommend to compile the kernel in 486 mode? I talked to someone who uses oracle on 2.4.x and he told me it is even in the latest docs. What is the voodoo behind that? Btw he has no freezes or the like, but occasional coredumps from oracle processes, which he states as "not nice, but no showstopper" as his clients reconnect/retransmit with only a slight delay. This may be related to VM, thats why I will try to convince him of some patches :-) and have a look at the coredump-frequency.

Regards,
Stephan

That would do it, but I was trying to give a real-world example from image processing, like copying a large image file.

I use FFTW from http://www.fftw.org.

Imagine a 16384 by 16384 array of double complex values. That's a 4 GByte image. Scale down to fit your machine, of course :). The first pass will do an FFT on every row (column) if your language is C (FORTRAN). The "stride" is 16 bytes (one complex value) in the inner loop. Each row (column) is 16384*16 = 262144 bytes long, which works out to 64 pages if the page size is 4096 bytes.

Then the second pass will do an FFT on every column (row). The stride is 16384*16 = 262144 bytes. This is a new page for each 16-byte complex value you process :-). That is, all 16384 pages have to be in memory, or swapped into memory if you've run out of real memory and the kernel has swapped them out.

Please ... *don't* try to do this on a 512 MB machine and think that an efficient VM is gonna make it work :),
--
M. Edward Borasky

[email protected]
http://www.borasky-research.net

What phrase will you *never* hear Miss Piggy use?
"You can't make a silk purse out of a sow's ear!"

Image processing people use tiling. Try loading a giant image into the gimp and into a non smart application like xpaint. The difference is huge just by careful implementation of the algorithms

Yes but you don't do it that way, you do stripes of parallel fft computations. We can all write dumb programs that don't behave well with the VM layer.

Alan

You're right ... no one does an *out-of-core* 2D FFT using VM. What I am saying is that a large page cache can turn an *in-core* 2D FFT -- a 4 GB case on an 8 GB machine, for example -- into an out-of-core one!

One other data point: on my stock Red Hat 7.2 box with 512 MB of RAM, I ran a Perl script that builds a 512 MByte hash, a second Perl script which creates a 512 MByte disk file, and the check pass of FFTW concurrently. As I expected, the two Perl scripts competed for RAM and slowed down FFTW. What was even more interesting, though, was that the VM apparently functions correctly in this instance. All three of the processes were getting CPU cycles. And I never saw "kswapd" or "kupdated" take over the system.

Although the page cache did get large at one point, once the hash builder got to about 400 MBytes in size, the "cached" piece shrunk to about 10 MBytes and most of the RAM got allocated to the hash builder, as did appropriate amounts of swap. In short, the kernel in Red Hat 7.2 with under 1 GByte of memory is behaving well under memory pressure. It looks like it's kernels beyond that one that have the problems, and also systems with more than 1 GByte. If I had the money, I'd stuff some more RAM in the machine and see if I could isolate this a little further. If anyone wants my Perl scripts, which are trivial, let me know.
--
M. Edward Borasky
[email protected]
http://www.borasky-research.net

No big wonder if both -aa and -rmap only get tested without swap ;)

Rik

Shortwave goes a long way:  irc.starchat.net  #swl

http://www.surriel.com/		http://distro.conectiva.com/

To be clear ... -aa and -rmap should of course also work nicely without swap, no excuses for the bad behaviour shown in Martin's test, but at the moment they simply don't seem tuned for it.

regards,

Rik

Shortwave goes a long way:  irc.starchat.net  #swl

http://www.surriel.com/		http://distro.conectiva.com/

After some time, ranging from 1 to 48 hours, mysql quits in an unclean fashion (dies leaving tables improperly closed) with a dump in the mysql log file that looks like:

Which the Mysql support team says appears to be memory corruption. Since this has happened on 4 different machines, and one of them had memtest86 run on it (coming up clean), they seem (witness Sasha's post) to think this may have something to do with the memory handling in the kernel.

I haven't run it on a kernel that has debugging enabled yet, partially because I've been tracing a completely unrelated problems with our hard drives (IBM GXP 75G drives made in Hungary during the first 3 months of 2001), and partially because the only way to get this to happen is to put the database in production, which results in a crash, which takes our site offline, which costs us money and pisses off our users. Right now we're running on a sun e4500, and it's stable, so until we get the other problem worked out, we're waiting to see on this one.

Share and Enjoy. 

Good to hear we agree it _should_ work. When does it (rmap)? ;-)

Regards,
Stephan

mysql question: is this a binary from some distro or self-compiled? If self-compiled can you show your ./configure paras, please?

There is a big difference between memory _corruption_ and a VM deficiency. No app can cope with a _corruption_ and is perfectly allowed to core dump or exit (or trash your disk). But this should not happen on allocation failures.

Unless all your RAM is from the same series I do not really believe in mem corruption. I would try Martins small VM patch, as it looks like being a bit more efficient in low mem conditions and this may well be the case you are running into. This means 2.4.17 standard + patch.

Regards,
Stephan

It's the binary from mysql.com.

Is there a reasonable chance that martins patch will get mainlined in the near future? One of the big reasons I chose to upgrade to a later kernel version (from 2.4.8ac<something>+LVMpatches+...) was to get away from having to apply patches (and document which patches and where to get them etc).

If this is the route I have to go, I'll do it but, well, I'm not that comfortable with it.

Share and Enjoy. 

I integrated Ed Tomlinson's patch today and have made one more small change. In the patches I ran here things worked fine, the system avoids OOM now.

Problem is, it doesn't seem to want to run the OOM killer when needed, at least not any time soon. I need to check out this code again later.

Anyway, rmap-11 should work fine for your test. ;)

regards,

Rik

Shortwave goes a long way:  irc.starchat.net  #swl

http://www.surriel.com/		http://distro.conectiva.com/

self-compiled? If

Beta or stable release?

bit

you are

mainlined

I really can't know. But to me the results look interesting enough to give it a try on certain problem situations (like yours) to find out if it is any better than the stock version. If you and others can confirm that things get better then I have no real doubts that Marcelo can pick it up.

was

Well, there is really nothing wrong with upgrading mainline kernels, as the are getting better with every release, so I would always suggest to take the releases up lets say a week after being out. Only your situation maybe can help to improve more, if you input some of your experiences in LKML with a patch like Martins. Feedback _is_ required to find a solution to an existing problem.

not

Well, my suggestions: don't patch around too much, but try single patches on stock kernel and evaluate them here.

Regards,
Stephan

Stable.

Out of ignorance and laziness, where is it again that I can get this kernel?

Funny, I went from a working 2.4.8-ac<x> to a non-working 2.4.13+patches when I started getting these crashes. At first I thought they were Mysql, so I called them. They said "Re-install windows", er, I mean upgrade my kernel to 2.4.16, which would "fix the problem", so I did, and it didn't. So they said to go to 2.4.17rc2 as that would fix my problem, only it didn't.

Yeah, and build a debian package, distribute it to (looks behind me) 100+ linux servers, including 6 mission critical heavily loaded DB machines.

Not to be a complete asswipe, but no. While I like playing with computers and all that, I don't have enough hours in the day to be rolling out new kernels every couple weeks and still have time left over to see my wife, shoot my guns, ride my motorcycles and drink my scotch.

I understand completely, I'm just trying to figure out a way to test this that doesn't impact my site as drastically. See, we've only got two databases that will cause this fault, and of course they are the two most important ones, and the only way we can generate this fault is to put them live and wait for them to crash.

There are 2 other patches I need to apply, the first is the LVM 1.0.1 patch, and the second is the VFS-lock patch. We need these to do snapshots. Which isn't bad, but I'm about the only one still here who can do it (violates hit-by-a-bus rule).

Share and Enjoy. 

On Sun, Jan 06, 2002 at 04:48:13PM +0100, Stephan von Krawczynski wrote:
[...]

| I read all your LKML mails since beginning of November, could find a lot about
| cpu, configs,tops etc but not a single "cat /proc/interrupts" together with
| uptime.

http://web.irridia.com/info/linux/APIC/

This was published back in the beginning (4/2001), and additional stuff sent to Alan and Manfred for debugging. I was pushing my problem on LKML for a couple of weeks, but without much feedback I'm sticking to my workaround.

This also feeds back to my earlier thoughts on some kind of LKML summary page of patches and problem reports for those disinclined to wade through the high LKML traffic. It's hard for me, much less you, to go back through the archives manually...

| According to the ongoings of your mails you seem to try really a lot of things
| to make it work out. I recommend not to intermix the patches a lot. I would
| stay close to marcelo's tree and try _single_ small patches on top of that. If
| you mix them up (even only two of them) you won't be able to track down very
| well, what is really better or worse.

Actually, that's why I don't test -aa. Whatever Marcelo chooses to include, I'll trust it in its entirety. But I've tested, for example, Linus' locked memory patch, and a couple of Andrew's isolated patches, all applied to mainline with nothing else. I can't try -aa because it has interdependencies and unintentional (I assume) backouts of code.

| One thing I would like to ask here is this (as you are dealing with oracle
| stuff): why does oracle recommend to compile the kernel in 486 mode? I talked
| to someone who uses oracle on 2.4.x and he told me it is even in the latest
| docs. What is the voodoo behind that? Btw he has no freezes or the like, but
| occasional coredumps from oracle processes, which he states as "not nice, but
| no showstopper" as his clients reconnect/retransmit with only a slight delay.
| This may be related to VM, thats why I will try to convince him of some patches
| :-) and have a look at the coredump-frequency.

I haven't had any problems with Oracle at all since Linus' locked memory patch back in the 2.4.14-15ish days. This on a 4GB 6-way Xeon with ext2, reiser, couple of other complications, with the kernel compiled for P3. I really don't know what would cause Oracle to misbehave with an i686 kernel that wouldn't be a kernel bug.

Perhaps a gcc-related bug? I'm still using 2.91.66 for kernels, although I've used 2.95.x with no problems. I'm not touching 2.96.x with a ten-foot pole, waiting instead for a sane 3.x one of these years.

I think Oracle (the company) is a little short of tooth on Linux experience, since for example and AFAIK they never discovered the fatal 2.4 locked memory problem -- that took Google's report and to a much lesser extent my later discovery of the same problem.

Ken.
[email protected]

Let me rephrase that.

Out of ignorance and laziness, exactly which patch is it that I need, and where can I find it?

Share and Enjoy. 

I stayed at work all night banging out tests on a few of our machines here. I took 2.4.18-pre2 and 2.4.18-pre2 with the vmscan patch from "M.H.VanLeeuwen" <[email protected]>.

My sustained test consisted of this type of load:

	ls -lR / > /dev/null &
	/usr/bin/slocate -u -f "nfs,smbfs,ncpfs,proc,devpts" -e "/tmp,/var/tmp,/usr/tmp,/afs,/net" &
	dd if=/dev/sda3 of=/sda3 bs=1024k &
	# Hit TUX on this machine repeatedly; html page with 1000 images
	# Wait for memory to be mostly used by buff/page cache
	./a.out &
	# repeat finished commands -- keep all commands running
	# after a.out finishes, alow buff/page to refill before repeating

The a.out in this case is a little program (attached, c.c) to allocate and write to an amount of memory equal to physical RAM. The example I chose below is from a 2xP3/600 with 1GB of RAM and 2GB swap.

This was not a formal benchmark -- I think benchmarks have been presented before by other folks, and looking at benchmarks does not necessarily indicate the real-world problems that exist. My intent was to reproduce the issues I've been seeing, and then apply the MH (and only the MH) patch and observe.

2.4.18-pre2

Once slocate starts and gets close to filling RAM with buffer/page cache, kupdated and kswapd have periodic spikes of 50-100% CPU.

When a.out starts, kswapd and kupdated begin to eat significant portions of CPU (20-100%) and I/O becomes more and more sluggish as a.out allocates.

When a.out uses all free RAM and should begin eating cache, significant swapping begins and cache is not decreased significantly until the machine goes 100-200MB into swap.

Here are two readprofile outputs, sorted by ticks and load.

229689 default_idle 4417.0962

  4794 file_read_actor                           18.4385
   405 __rdtsc_delay                             14.4643
  3763 do_anonymous_page                         14.0410
  3796 statm_pgd_range                            9.7835
  1535 prune_icache                               6.9773
   153 __free_pages                               4.7812
  1420 create_bounce                              4.1765
   583 sym53c8xx_intr                             3.9392
   221 atomic_dec_and_lock                        2.7625
  5214 generic_file_write                         2.5659

273464 total 0.1903
234168 default_idle 4503.2308
5298 generic_file_write 2.6073
4868 file_read_actor 18.7231
3799 statm_pgd_range 9.7912
3763 do_anonymous_page 14.0410

  1535 prune_icache                               6.9773
  1526 shrink_cache                               1.6234
  1469 create_bounce                              4.3206
   643 rmqueue                                    1.1320
   591 sym53c8xx_intr                             3.9932
   505 __make_request                             0.2902

2.4.18-pre2 with MH

With the MH patch applied, the issues I witnessed above did not seem to reproduce. Memory allocation under pressure seemed faster and smoother. kswapd never went above 5-15% CPU. When a.out allocated memory, it did not begin swapping until buffer/page cache had been nearly completely cannibalized.

And when a.out caused swapping, it was controlled and behaved like you would expect the VM to bahave -- slowly swapping out unused pages instead of large swap write-outs without the patch.

Martin, have you done throughput benchmarks with MH/rmap/aa, BTW?

But both kernels still seem to be sluggish when it comes to doing small I/O operations (vi, ls, etc) during heavy swapping activity.

Here are the readprofile results:

206243 default_idle 3966.2115

  6486 file_read_actor                           24.9462
   409 __rdtsc_delay                             14.6071
  2798 do_anonymous_page                         10.4403
   185 __free_pages                               5.7812
  1846 statm_pgd_range                            4.7577
   469 sym53c8xx_intr                             3.1689
   176 atomic_dec_and_lock                        2.2000
   349 end_buffer_io_async                        1.9830
   492 refill_inactive                            1.8358
    94 system_call                                1.8077

245776 total 0.1710
216238 default_idle 4158.4231
6486 file_read_actor 24.9462
2799 do_anonymous_page 10.4440

  1855 statm_pgd_range                            4.7809
  1611 generic_file_write                         0.7928
   839 __make_request                             0.4822
   820 shrink_cache                               0.7374
   540 rmqueue                                    0.9507
   534 create_bounce                              1.5706
   492 refill_inactive                            1.8358
   487 sym53c8xx_intr                             3.2905

There may be significant differences in the profile outputs for those with VM fu.

Summary: MH swaps _after_ cache has been properly cannibalized, and swapping activity starts when expected and is properly throttled. kswapd and kupdated don't seem to go into berserk 100% CPU mode.

At any rate, I now have the MH patch (and Andrew Morton's mini-ll and read-latency2 patches) in production, and I like what I see so far. I'd vote for them to go into 2.4.18, IMHO. Maybe the full low-latency patch if it's not truly 2.5 material.

My next cook-off will be with -aa and rmap, although if the rather small MH patch fixes my last issues it may be worth putting all VM effort into a 2.5 VM cook-off. :) Hopefully the useful stuff in -aa can get pulled in at some point soon, though.

Thanks much to Martin H. VanLeeuwen for his patch and Stephan von Krawczynski for his recommendations. I'll let MH cook for a while and I'll follow up later.

Ken.
[email protected]

c.c:

#include <stdio.h>

#define MB_OF_RAM 1024

int
main()
{
	long stuffsize = MB_OF_RAM * 1048576 ;
	char *stuff ;

	if ( stuff = (char *)malloc( stuffsize ) ) {
		long chunksize = 1048576 ;
		long c ;

		for ( c=0 ; c<chunksize ; c++ )
			*(stuff+c) = '\0' ;
		/* hack; last chunk discarded if stuffsize%chunksize != 0 */
		for ( ; (c+chunksize)<stuffsize ; c+=chunksize )
			memcpy( stuff+c, stuff, chunksize );
	
		sleep( 120 );
	}
	else
		printf("OOPS\n");

	exit( 0 );
}

On Sat, Jan 05, 2002 at 04:08:33PM +0100, Stephan von Krawczynski wrote:
[...]

| I am pretty impressed by Martins test case where merely all VM patches fail
| with the exception of his own :-) The thing is, this test is not of nature
| "very special" but more like "system driven to limit by normal processes". And
| this is the real interesting part about it.

[...]

After more testing, my original observations seem to be holding up, except that under heavy VM load (e.g., "make -j bzImage") the machine's overall performance seems far lower. For instance, without the patch the -j build finishes in ~10 minutes (2x933P3/256MB) but with the patch I haven't had the patience to let it finish after more than an hour.

This is perhaps because the vmscan patch is too aggressively shrinking the caches, or causing thrashing in another area? I'm also noticing that the amount of swap used is nearly an order of magnitude higher, which doesn't make sense at first glance... Also, there are extended periods where idle CPU is 50-80%.

Maybe the patch or at least its intent can be merged with Andrea's work if applicable?

Thanks,

Ken.
[email protected]

On Thu, Jan 03, 2002 at 02:23:01PM -0600, Ken Brownfield wrote:

| Unfortunately, I lost the response that basically said "2.4 looks stable
| to me", but let me count the ways in which I agree with Andreas'
| sentiment:
| 
| 	A) VM has major issues
| 		1) about a dozen recent OOPS reports in VM code
| 		2) VM falls down on large-memory machines with a
| 		   high inode count (slocate/updatedb, i/dcache)
| 		3) Memory allocation failures and OOM triggers
| 		   even though caches remain full.
| 		4) Other bugs fixed in -aa and others
| 	B) Live- and dead-locks that I'm seeing on all 2.4 production
| 	   machines > 2.4.9, possibly related to A.  But how will I
| 	   ever find out?
| 	C) IO-APIC code that requires noapic on any and all SMP
| 	   machines that I've ever run on.
| 
| I don't have anything against anyone here -- I think everyone is doing a
| fine job.  It's an issue of acceptance of the problem and focus.  These
| issues are all showstoppers for me, and while I don't represent the 90%
| of the Linux market that is UP desktops, IMHO future work on the kernel
| will be degraded by basic functionality that continues to cause
| problems.
| 
| I think seeing some of Andrea's and Andrew's et al patches actually
| *happen* would be a good thing, since 2.4 kernels are decidedly not
| ready for production here.  I am forced to apply 26 distinct patch sets
| to my kernels, and I am NOT the right person to make these judgements.
| Which is why I was interested in an LKML summary source, though I
| haven't yet had a chance to catch up on that thread of comment.
| 
| Having a glitch in the radeon driver is one thing; having persistent,
| fatal, and reproducable failures in universal kernel code is entirely
| another.
| 
| -- 
| Ken.
| [email protected]
| 
| 
| On Fri, Dec 28, 2001 at 09:16:38PM +0100, Andreas Hartmann wrote:
| | Hello all,
| | 
| | Again, I did a rsync-operation as described in
| | "[2.4.17rc1] Swapping" MID <[email protected]>.
| | 
| | This time, the kernel had a swappartition which was about 200MB. As the 
| | swap-partition was fully used, the kernel killed all processes of knode.
| | Nearly 50% of RAM had been used for buffers at this moment. Why is there 
| | so much memory used for buffers?
| | 
| | I know I repeat it, but please:
| | 
| | 	Fix the VM-management in kernel 2.4.x. It's unusable. Believe
| | 	me! As comparison: kernel 2.2.19 didn't need nearly any swap for
| | 	the same operation!
| | 
| | Please consider that I'm using 512 MB of RAM. This should, or better: 
| | must be enough to do the rsync-operation nearly without any swapping - 
| | kernel 2.2.19 does it!
| | 
| | The performance of kernel 2.4.18pre1 is very poor, which is no surprise, 
| | because the machine swaps nearly nonstop.
| | 
| | 
| | Regards,
| | Andreas Hartmann
| | 
| | -
| | To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
| | the body of a message to [email protected]
| | More majordomo info at  http://vger.kernel.org/majordomo-info.html
| | Please read the FAQ at  http://www.tux.org/lkml/
| -
| To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
| the body of a message to [email protected]
| More majordomo info at  http://vger.kernel.org/majordomo-info.html
| Please read the FAQ at  http://www.tux.org/lkml/

please, PLEASE stop using "make -j"
for anything except the fork-bomb that it is. pretending that it's a benchmark, especially one to guide kernel tuning, is a travesty!

if you want to simulate VM load, so something sane like boot with mem=32M, or a simple "mmap(lots); mlockall" tool.

regards, mark hahn.

Andrew Morton kindly pointed out that my crack pipe is dangerously empty and I didn't specify what patch I was talking about. In my defense, I was up all last night tracking down the ext3 bug that Andrew fixed right under me. ;)

I replied to the wrong message, which I've pasted below. This is wrt Martin's VM patch per the previous discussion.

Apologies,

Ken.
[email protected]

On Fri, Jan 11, 2002 at 02:41:17PM -0600, Ken Brownfield wrote:

| After more testing, my original observations seem to be holding up,
| except that under heavy VM load (e.g., "make -j bzImage") the machine's
| overall performance seems far lower.  For instance, without the patch
| the -j build finishes in ~10 minutes (2x933P3/256MB) but with the patch
| I haven't had the patience to let it finish after more than an hour.
| 
| This is perhaps because the vmscan patch is too aggressively shrinking
| the caches, or causing thrashing in another area?  I'm also noticing
| that the amount of swap used is nearly an order of magnitude higher,
| which doesn't make sense at first glance...  Also, there are extended
| periods where idle CPU is 50-80%.
| 
| Maybe the patch or at least its intent can be merged with Andrea's work
| if applicable?
| 
| Thanks,
| -- 
| Ken.
| [email protected]

What I SHOULD have replied to:

| Date:   Tue, 8 Jan 2002 09:19:57 -0600
| From: Ken Brownfield <[email protected]>
| To: Stephan von Krawczynski <[email protected]>,
|         "M.H.VanLeeuwen" <[email protected]>, [email protected]
| Cc: [email protected]
| Subject: Update Re: [2.4.17/18pre] VM and swap - it's really unusable
| User-Agent: Mutt/1.2.5.1i
| In-Reply-To: <[email protected]>; from [email protected] o
| n Sat, Jan 05, 2002 at 04:08:33PM +0100
| Precedence: bulk
| X-Mailing-List:         [email protected]
| 
| I stayed at work all night banging out tests on a few of our machines
| here.  I took 2.4.18-pre2 and 2.4.18-pre2 with the vmscan patch from
| "M.H.VanLeeuwen" <[email protected]>.
| 
| My sustained test consisted of this type of load:
| 
|         ls -lR / > /dev/null &
|         /usr/bin/slocate -u -f "nfs,smbfs,ncpfs,proc,devpts" -e "/tmp,/var/tmp,/
| usr/tmp,/afs,/net" &
|         dd if=/dev/sda3 of=/sda3 bs=1024k &
|         # Hit TUX on this machine repeatedly; html page with 1000 images
|         # Wait for memory to be mostly used by buff/page cache
|         ./a.out &
|         # repeat finished commands -- keep all commands running
|         # after a.out finishes, alow buff/page to refill before repeating
| 
| The a.out in this case is a little program (attached, c.c) to allocate
| and write to an amount of memory equal to physical RAM.  The example I
| chose below is from a 2xP3/600 with 1GB of RAM and 2GB swap.
| 
| This was not a formal benchmark -- I think benchmarks have been
| presented before by other folks, and looking at benchmarks does not
| necessarily indicate the real-world problems that exist.  My intent was
| to reproduce the issues I've been seeing, and then apply the MH (and
| only the MH) patch and observe.
| 
| 2.4.18-pre2
| 
| Once slocate starts and gets close to filling RAM with buffer/page
| cache, kupdated and kswapd have periodic spikes of 50-100% CPU.
| 
| When a.out starts, kswapd and kupdated begin to eat significant portions
| of CPU (20-100%) and I/O becomes more and more sluggish as a.out
| allocates.
| 
| When a.out uses all free RAM and should begin eating cache, significant
| swapping begins and cache is not decreased significantly until the
| machine goes 100-200MB into swap.
| 
| Here are two readprofile outputs, sorted by ticks and load.
| 
| 229689 default_idle                             4417.0962
|   4794 file_read_actor                           18.4385
|    405 __rdtsc_delay                             14.4643
|   3763 do_anonymous_page                         14.0410
|   3796 statm_pgd_range                            9.7835
|   1535 prune_icache                               6.9773
|    153 __free_pages                               4.7812
|   1420 create_bounce                              4.1765
|    583 sym53c8xx_intr                             3.9392
|    221 atomic_dec_and_lock                        2.7625
|   5214 generic_file_write                         2.5659
| 
| 273464 total                                      0.1903
| 234168 default_idle                             4503.2308
|   5298 generic_file_write                         2.6073
|   4868 file_read_actor                           18.7231
|   3799 statm_pgd_range                            9.7912
|   3763 do_anonymous_page                         14.0410
|   1535 prune_icache                               6.9773
|   1526 shrink_cache                               1.6234
|   1469 create_bounce                              4.3206
|    643 rmqueue                                    1.1320
|    591 sym53c8xx_intr                             3.9932
|    505 __make_request                             0.2902
| 
| 
| 2.4.18-pre2 with MH
| 
| With the MH patch applied, the issues I witnessed above did not seem to
| reproduce.  Memory allocation under pressure seemed faster and smoother.
| kswapd never went above 5-15% CPU.  When a.out allocated memory, it did
| not begin swapping until buffer/page cache had been nearly completely
| cannibalized.
| 
| And when a.out caused swapping, it was controlled and behaved like you
| would expect the VM to bahave -- slowly swapping out unused pages
| instead of large swap write-outs without the patch.
| 
| Martin, have you done throughput benchmarks with MH/rmap/aa, BTW?
| 
| But both kernels still seem to be sluggish when it comes to doing small
| I/O operations (vi, ls, etc) during heavy swapping activity.
| 
| Here are the readprofile results:
| 
| 206243 default_idle                             3966.2115
|   6486 file_read_actor                           24.9462
|    409 __rdtsc_delay                             14.6071
|   2798 do_anonymous_page                         10.4403
|    185 __free_pages                               5.7812
|   1846 statm_pgd_range                            4.7577
|    469 sym53c8xx_intr                             3.1689
|    176 atomic_dec_and_lock                        2.2000
|    349 end_buffer_io_async                        1.9830
|    492 refill_inactive                            1.8358
|     94 system_call                                1.8077
| 
| 245776 total                                      0.1710
| 216238 default_idle                             4158.4231
|   6486 file_read_actor                           24.9462
|   2799 do_anonymous_page                         10.4440
|   1855 statm_pgd_range                            4.7809
|   1611 generic_file_write                         0.7928
|    839 __make_request                             0.4822
|    820 shrink_cache                               0.7374
|    540 rmqueue                                    0.9507
|    534 create_bounce                              1.5706
|    492 refill_inactive                            1.8358
|    487 sym53c8xx_intr                             3.2905
| 
| 
| There may be significant differences in the profile outputs for those
| with VM fu.  
| 
| Summary: MH swaps _after_ cache has been properly cannibalized, and
| swapping activity starts when expected and is properly throttled.
| kswapd and kupdated don't seem to go into berserk 100% CPU mode.
| 
| At any rate, I now have the MH patch (and Andrew Morton's mini-ll and
| read-latency2 patches) in production, and I like what I see so far.  I'd
| vote for them to go into 2.4.18, IMHO.  Maybe the full low-latency patch
| if it's not truly 2.5 material.
| 
| My next cook-off will be with -aa and rmap, although if the rather small
| MH patch fixes my last issues it may be worth putting all VM effort into
| a 2.5 VM cook-off. :)  Hopefully the useful stuff in -aa can get pulled
| in at some point soon, though.
| 
| Thanks much to Martin H. VanLeeuwen for his patch and Stephan von
| Krawczynski for his recommendations.  I'll let MH cook for a while and
| I'll follow up later.
| -- 
| Ken.
| [email protected]
| 
| c.c:
| 
| #include <stdio.h>
| 
| #define MB_OF_RAM 1024
| 
| int
| main()
| {
|         long stuffsize = MB_OF_RAM * 1048576 ;
|         char *stuff ;
| 
|         if ( stuff = (char *)malloc( stuffsize ) ) {
|                 long chunksize = 1048576 ;
|                 long c ;
| 
|                 for ( c=0 ; c<chunksize ; c++ )
|                         *(stuff+c) = '\0' ;
|                 /* hack; last chunk discarded if stuffsize%chunksize != 0 */
|                 for ( ; (c+chunksize)<stuffsize ; c+=chunksize )
|                         memcpy( stuff+c, stuff, chunksize );
|         
|                 sleep( 120 );
|         }
|         else
|                 printf("OOPS\n");
| 
|         exit( 0 );
| }
| 
| 
| On Sat, Jan 05, 2002 at 04:08:33PM +0100, Stephan von Krawczynski wrote:
| [...]
| | I am pretty impressed by Martins test case where merely all VM patches fail
| | with the exception of his own :-) The thing is, this test is not of nature
| | "very special" but more like "system driven to limit by normal processes". And
| | this is the real interesting part about it.
| [...]
| 

I don't think I made the claim that this was a benchmark -- I certainly realize that "make -j bzImage" is not real-world, but it is clearly indicative of heavy VM/CPU/context load. Since I don't believe this patch is currently in the running for inclusion, I'm just giving general feedback to the patch author rather than making a case.

For instance, "make -j bzImage" reproduced the ext3 bug that Andrew found where my other VM-intensive apps did not. I doubt we should keep the bug in the kernel because the situation isn't real-world enough.

But yes, a bug is worse than a behavior flaw, granted.

Ken.
[email protected]

On Fri, Jan 11, 2002 at 04:13:00PM -0500, Mark Hahn wrote:

| > overall performance seems far lower.  For instance, without the patch
| > the -j build finishes in ~10 minutes (2x933P3/256MB) but with the patch
| 
| please, PLEASE stop using "make -j" 
| for anything except the fork-bomb that it is.
| pretending that it's a benchmark, especially one 
| to guide kernel tuning, is a travesty!
| 
| if you want to simulate VM load, so something sane like
| boot with mem=32M, or a simple "mmap(lots); mlockall" tool.
| 
| regards, mark hahn.
| 
| -
| To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
| the body of a message to [email protected]
| More majordomo info at  http://vger.kernel.org/majordomo-info.html
| Please read the FAQ at  http://www.tux.org/lkml/

Actually, it's as good a benchmark as any. Knowing how well the system is able to recover from heavy overload situations is useful to know if your server gets heavily overloaded at times.

If one VM falls over horribly under half the load it takes to make another VM go slower, I know which one I'd want on my server.

... and then you come up with something WAY less realistic than 'make -j' ;)))

cheers,

Rik

"Linux holds advantages over the single-vendor commercial OS"
    -- Microsoft's "Competing with Linux" document

http://www.surriel.com/		http://distro.conectiva.com/

Ken,

Attached is an update to my previous vmscan.patch.2.4.17.c

Version "d" fixes a BUG due to a race in the old code _and_ is much less agressive at cache_shrinkage or conversely more willing to swap out but not as much as the stock kernel.

It continues to work well wrt to high vm pressure.

Give it a whirl to see if it changes your "-j" symptoms.

If you like you can change the one line in the patch from "DEF_PRIORITY" which is "6" to progressively smaller values to "tune" whatever kind of swap_out behaviour you like.

Martin