kswapd using all cpu for long periods in 2.4.9-pre4

Hi,
Somewhere between 2.4.8 and 2.4.9-pre4 kswapd developed a problem. Actually it looks like it is related to the changes in do_try_to_free_pages...

kswapd is using all cpu for long periods (100-200 seconds then 100-200 seconds break....) there is very little disk activity (heres a vmstat while its happening)

0 0 1 18372 1872 1628 38540 1 0 4 2 343 127 1 96 3 0 0 1 18360 1992 1628 38416 0 0 0 3 282 120 0 100 0 0 0 1 18312 2188 1632 38244 1 0 31 6 376 147 1 89 10 0 0 1 18324 1908 1632 38252 0 0 0 3 411 130 0 99 0 0 0 1 18324 1996 1632 38260 0 0 1 1 175 115 0 100 0 0 0 1 18324 1852 1632 38336 17 0 36 8 150 163 0 100 0 0 0 1 18324 2028 1632 38336 0 0 0 8 107 119 0 100 0

kswapd task:
Trace; c0145328 <shrink_icache_memory+18/30>
Trace; c012b850 <do_try_to_free_pages+60/c0>
Trace; c012b90e <kswapd+5e/c0>
Trace; c0105000 <_stext+0/0>
Trace; c0105000 <_stext+0/0>
Trace; c01054d6 <kernel_thread+26/30>
Trace; c012b8b0 <kswapd+0/c0>

meminfo just before kswapd goes crazy:
Mem-info:
Free pages: 4616kB ( 0kB HighMem)
( Active: 6707, inactive_dirty: 4312, inactive_clean: 895, free: 1154 (256 512 2048) )
5*4kB 1*8kB 1*16kB 1*32kB 1*64kB 1*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB = 1036kB)
483*4kB 104*8kB 5*16kB 1*32kB 1*64kB 1*128kB 0*256kB 1*512kB 0*1024kB 0*2048kB = 3580kB)
= 0kB)
Swap cache: add 16566, delete 14047, find 17323/32269
Free swap: 110016kB
16368 pages of RAM
0 pages of HIGHMEM
689 reserved pages
10205 pages shared
2519 pages swap cached
0 pages in page table cache
Buffer memory: 1472kB

mem info during:
SysRq: Show Memory
Mem-info:
Free pages: 3744kB ( 0kB HighMem)
( Active: 6946, inactive_dirty: 4296, inactive_clean: 895, free: 936 (256 512 2048) )
5*4kB 1*8kB 1*16kB 1*32kB 1*64kB 1*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB = 1036kB)
263*4kB 105*8kB 5*16kB 1*32kB 1*64kB 1*128kB 0*256kB 1*512kB 0*1024kB 0*2048kB = 2708kB)
= 0kB)
Swap cache: add 16680, delete 14047, find 17408/32591
Free swap: 110016kB
16368 pages of RAM
0 pages of HIGHMEM
689 reserved pages
10453 pages shared
2633 pages swap cached
0 pages in page table cache
Buffer memory: 1480kB

In article <000c01c12569$65581630$0200a8c0@W2K> you write:

For some reason do_try_to_free_pages() doesn't end up ever being happy with how much memory there is free (that's why kswapd will loop). But you actually have a reasonable amount of free memory, so all the memory free'ers will actually refuse to do anything. The problem seems to be that either inactive_shortage() or free_shortage() will just continually claim that you have a shortage of memory.

However, you actually do have memory:

I bet this is it.

It probably decides that you have a shortage of HIGHMEM pages. Which is understandable, since you don't have any ;)

Does the following trivial patch fix it for you?

(If it doesn't, please try just removing the "continue" at line 930 or so in the middle of the kswapd loop. But I think there's something else that makes kswapd just think it _should_ loop, and the lack of zone->size testing in the inactive_shortage() calculations looks like it).

Thanks,

Linus

-----

@@ -788,6 +788,9 @@
 			zone_t *zone = pgdat->node_zones + i;
 			unsigned int inactive;
 
+			if (!zone->size)
+				continue;
+
 			inactive  = zone->inactive_dirty_pages;
 			inactive += zone->inactive_clean_pages;
 			inactive += zone->free_pages;

I just noticed something..

In article <000c01c12569$65581630$0200a8c0@W2K> you write:

This shows your freepages.min low and high: 256. 512 and 2048 respectively.

And that's WRONG. Your "freepages.high" is _way_ too high. You must have changed it through the /proc interface somehow, which should be impossible (the /proc/sys/vm/freepages file is read-only), but I suspect that maybe there's a sysctl backdoor.

How do I know that "freepages.high" is too big? Because it should always be 3*freepages.min. Yours is 8*min.

And why does this matter? Because what has apparently happened is:
- your setup somehow modifies the global free target
- but the per-zone targets are still the old ones

The end result: we always think that we are under a global shortage, but the actual page freeing routines refuse to free pages because all the _zones_ think that they are fine.

Which gives you _exactly_ the behaviour you're describing. "kswapd" will run forever, and never make any progress at all.

I wonder if this is more common? Does any of the distributions maybe have some "system tuning" script that modifies the freepages values through sysctl? That would certainly explain why some people see the problem and others do not..

How DID you change that value?

Linus

Yes you're right, my fault sorry. Leaving it as default does fix the problem. The values were changed using sysctl(8). It writes straight to /proc - my freepages is not readonly for some reason (source has been changed)... anyway, no sysctl(2) backdoors found here.

On related topic - is /Documentation/sysctl/vm.txt up-to-date? Is there a better document to use?

Nick

----- Original Message -----
From: "Linus Torvalds" <[email protected]>
Newsgroups: linux.dev.kernel
To: <[email protected]>; <[email protected]>; "Marcelo Tosatti" <[email protected]>; "Alan Cox" <[email protected]>
Sent: Thursday, August 16, 2001 3:19 AM
Subject: Re: kswapd using all cpu for long periods in 2.4.9-pre4

while