What version of the kernel fixes these VM issues?

DISCUSSION Anwar P 6 participants 16 messages Aug 24, 2001 – Aug 25, 2001

Subsystems (5)

Memory management Page allocator MGLRU MM reclaim +1 more subsystems

lore

Share & tools

lore

Anwar P

author

2001-08-24 09:16 UTC #0 lore raw

Hi -

We have big system (8 processors, 8GB ram), running Oracle and this other ETL tool. Oracle is up and running all the time, and the ETL tool runs once a day. But everytime the ETL tool runs (along with Oracle), the system seems to run out of memory, and the server comes to a crawl, often with keyborad response in 10 to 15 minute intervals. We are currently using the 2.4.3-6 kernel that comes with Redhat 7.1.

We know that Oracle comsumes no more than 2GB of memory at peak usage, and the ETL tool itself consumes less than 1GB. But the ETL tool does process a whole bunch of text files (total about 6GB worth of), and it runs for about 2 hours. What happens is that while they are both running, the filesystem cache size increases progressively, and some time later, it begins swapping. We do have 16GB (2x RAM) of swap. And when it starts to swap, the server responds to keystrokes/commands randomly and appears dead for 10s of minutes. We know that together our applications do not need more than 4GB of RAM on this 8GB box, so it is the VM that is causing this unnecessary swapping by trying to use too much memory for filesystem cache.

So the first question is, is there any way I can limit the amount of memory used for FS cache ?

And the next one is, are there any (later) versions of the kernel that are more sane about what the maximum FS cache it should use is ? It is strange that the FS caching does not take into account the amount of physical RAM on the box. What is the point in doing FS caching when the end result is thrashing and the machine becomes unusable ?

Anwar.

Signature · 6 lines

_______________________________________________
FREE Personalized E-mail at Mail.com 
http://www.mail.com/?sr=signup 

Talk More, Pay Less with Net2Phone Direct(R), up to 1500 minutes free! 
http://www.net2phone.com/cgi-bin/link.cgi?143

Daniel Phillips

2001-08-24 13:12 UTC #1 lore raw

Quoted 7 lines · Anwar P

We have big system (8 processors, 8GB ram), running Oracle and this other ETL tool. Oracle is up and running all the time, and the ETL tool runs once a day. But everytime the ETL tool runs (along with Oracle), the system seems to run out of memory, and the server comes to a crawl, often with keyborad response in 10 to 15 minute intervals. We are currently using the 2.4.3-6 kernel that comes with Redhat 7.1.

Quoted 10 lines

We know that Oracle comsumes no more than 2GB of memory at peak usage, and the ETL tool itself consumes less than 1GB. But the ETL tool does process a whole bunch of text files (total about 6GB worth of), and it runs for about 2 hours. What happens is that while they are both running, the filesystem cache size increases progressively, and some time later, it begins swapping. We do have 16GB (2x RAM) of swap. And when it starts to swap, the server responds to keystrokes/commands randomly and appears dead for 10s of minutes. We know that together our applications do not need more than 4GB of RAM on this 8GB box, so it is the VM that is causing this unnecessary swapping by trying to use too much memory for filesystem cache.

There is no way your system should be going into swap under these conditions
- it's a bug. We have probably fixed this already.

Quoted 2 lines

So the first question is, is there any way I can limit the amount of memory used for FS cache ?

Um, no, sorry. This should not be necessary.

Quoted 2 lines

And the next one is, are there any (later) versions of the kernel that are more sane about what the maximum FS cache it should use is ?

Please try 2.4.9 and 2.4.8-ac10. If the system slows down, look in your logs and see if there are any "allocation failed" messages. Use top or do watch cat /proc/meminfo to be sure your system isn't going into swap, and please let us know what happens.

--
Daniel

Nicolas Pitre

2001-08-24 18:15 UTC #2 lore raw

Quoted 19 lines · Daniel Phillips

Quoted 7 lines · Anwar P
We have big system (8 processors, 8GB ram), running Oracle and this other ETL tool. Oracle is up and running all the time, and the ETL tool runs once a day. But everytime the ETL tool runs (along with Oracle), the system seems to run out of memory, and the server comes to a crawl, often with keyborad response in 10 to 15 minute intervals. We are currently using the 2.4.3-6 kernel that comes with Redhat 7.1.
Quoted 10 lines
We know that Oracle comsumes no more than 2GB of memory at peak usage, and the ETL tool itself consumes less than 1GB. But the ETL tool does process a whole bunch of text files (total about 6GB worth of), and it runs for about 2 hours. What happens is that while they are both running, the filesystem cache size increases progressively, and some time later, it begins swapping. We do have 16GB (2x RAM) of swap. And when it starts to swap, the server responds to keystrokes/commands randomly and appears dead for 10s of minutes. We know that together our applications do not need more than 4GB of RAM on this 8GB box, so it is the VM that is causing this unnecessary swapping by trying to use too much memory for filesystem cache.

Quoted 7 lines

There is no way your system should be going into swap under these conditions
- it's a bug. We have probably fixed this already.
Please try 2.4.9 and 2.4.8-ac10. If the system slows down, look in your logs and see if there are any "allocation failed" messages. Use top or do watch cat /proc/meminfo to be sure your system isn't going into swap, and please let us know what happens.

I have a totally different setup but I can reproduce the same behavior on the system I have here:

ARM board with 32 MB RAM, no flash, NFS root.
The kernel is based on 2.4.8-ac9 plus some small VM fixes from -ac10.

My test consist in compiling gcc 3.0 while some MP3s are continously playing in the background. The gcc build goes pretty far along until both the mp3 player and the gcc build completely jam. Oh maybe not completely as I get about 100ms of audio playing every 10 secs. bash starts echoing what I type one char per approx 5 sec. The only thing that still works fine is the magic sysrq that clearly shows that the CPU is spinning in the VM code. NFS trafic is also going on full bandwidth but no progress ever happens in user space.

My console is on a serial port so if someone can send me a patch with whatever printks to trace what's happening in real time I'll be glad to provide the trace file. Reaching the jam state takes about 5 minutes so it's not hard to reproduce.

Nicolas

Nicolas Pitre

2001-08-24 18:27 UTC #3 lore raw

Quoted 4 lines · Nicolas Pitre

I have a totally different setup but I can reproduce the same behavior on the system I have here:
ARM board with 32 MB RAM, no flash, NFS root.

Sorry I meant no swap.

Nicolas

Mark Frazer

2001-08-24 18:48 UTC #4 lore raw

In reply to Nicolas Pitre

Hi Nicolas. You should run vmstat and watch the paging activity. We've seen our ARM boards get driven to a standstill by paging when they run out of RAM. We have no swap either. To make things worse, our flash loads are compressed, so we burn all our CPU in decompression when paging back in. If you have no swap, the only thing that can be booted out are executable pages.

Kernel folk: do /proc/sys/vm/{pagecache,buffermem} still do anything? Could Nicolas still limit the pagecache using these?

cheers
-mark

Nicolas Pitre <[email protected]> [01/08/24 14:38]:

Quoted 19 lines · Nicolas Pitre

Quoted 5 lines
I have a totally different setup but I can reproduce the same behavior on the system I have here:
ARM board with 32 MB RAM, no flash, NFS root.
Sorry I meant no swap.
Nicolas
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Daniel Phillips

2001-08-24 19:47 UTC #5 lore raw

In reply to Nicolas Pitre

Quoted 7 lines · Nicolas Pitre

Quoted 4 lines · Daniel Phillips
Please try 2.4.9 and 2.4.8-ac10. If the system slows down, look in your logs and see if there are any "allocation failed" messages. Use top or do watch cat /proc/meminfo to be sure your system isn't going into swap, and please let us know what happens.

Quoted 5 lines

I have a totally different setup but I can reproduce the same behavior on the system I have here:
ARM board with 32 MB RAM, no flash, NFS root.
The kernel is based on 2.4.8-ac9 plus some small VM fixes from -ac10.

What happens with 2.4.9?

Quoted 9 lines

My test consist in compiling gcc 3.0 while some MP3s are continously playing in the background. The gcc build goes pretty far along until both the mp3 player and the gcc build completely jam. Oh maybe not completely as I get about 100ms of audio playing every 10 secs. bash starts echoing what I type one char per approx 5 sec. The only thing that still works fine is the magic sysrq that clearly shows that the CPU is spinning in the VM code. NFS trafic is also going on full bandwidth but no progress ever happens in user space.

Quoted 4 lines

My console is on a serial port so if someone can send me a patch with whatever printks to trace what's happening in real time I'll be glad to provide the trace file. Reaching the jam state takes about 5 minutes so it's not hard to reproduce.

Try:

watch cat /proc/meminfo

--
Daniel

Daniel Phillips

2001-08-24 19:50 UTC #6 lore raw

In reply to Nicolas Pitre

Quoted 9 lines · Nicolas Pitre

I have a totally different setup but I can reproduce the same behavior on the system I have here:
ARM board with 32 MB RAM, no flash, NFS root.
The kernel is based on 2.4.8-ac9 plus some small VM fixes from -ac10.
My test consist in compiling gcc 3.0 while some MP3s are continously playing in the background. The gcc build goes pretty far along until both the mp3 player and the gcc build completely jam.

Which sound system, and which sound card driver?

Quoted 6 lines

Oh maybe not completely as I get
about 100ms of audio playing every 10 secs. bash starts echoing what I type one char per approx 5 sec. The only thing that still works fine is the magic sysrq that clearly shows that the CPU is spinning in the VM code. NFS trafic is also going on full bandwidth but no progress ever happens in user space.

--
Daniel

Nicolas Pitre

2001-08-24 20:13 UTC #7 lore raw

Quoted 4 lines · Daniel Phillips

Quoted 3 lines · Nicolas Pitre
I have a totally different setup but I can reproduce the same behavior on the system I have here:

Quoted 8 lines

Quoted 6 lines
ARM board with 32 MB RAM, no flash, NFS root.
The kernel is based on 2.4.8-ac9 plus some small VM fixes from -ac10.
My test consist in compiling gcc 3.0 while some MP3s are continously playing in the background. The gcc build goes pretty far along until both the mp3 player and the gcc build completely jam.
Which sound system, and which sound card driver?

The driver is for the UDA1341 on a SA1110 chip written by myself. It is fully OSS compliant, no ALSA.

Nicolas

Daniel Phillips

2001-08-24 22:29 UTC #8 lore raw

Quoted 9 lines · Nicolas Pitre

Quoted 7 lines · Daniel Phillips
Quoted 6 lines · Nicolas Pitre
I have a totally different setup but I can reproduce the same behavior on the system I have here:
ARM board with 32 MB RAM, no flash, NFS root.
The kernel is based on 2.4.8-ac9 plus some small VM fixes from -ac10.

Quoted 8 lines

Quoted 5 lines
Quoted 3 lines
My test consist in compiling gcc 3.0 while some MP3s are continously playing in the background. The gcc build goes pretty far along until both the mp3 player and the gcc build completely jam.
Which sound system, and which sound card driver?
The driver is for the UDA1341 on a SA1110 chip written by myself. It is fully OSS compliant, no ALSA.

Your system should be able to handle that easily. Do you have some meminfo output to look at? What about 2.4.9?

--
Daniel

Nicolas Pitre

2001-08-25 03:00 UTC #9 lore raw

Quoted 14 lines · Daniel Phillips

Quoted 13 lines · Nicolas Pitre
Quoted 11 lines · Daniel Phillips
Quoted 9 lines · Nicolas Pitre
I have a totally different setup but I can reproduce the same behavior on the system I have here:
ARM board with 32 MB RAM, no cache, NFS root.
The kernel is based on 2.4.8-ac9 plus some small VM fixes from -ac10.
My test consist in compiling gcc 3.0 while some MP3s are continously playing in the background. The gcc build goes pretty far along until both the mp3 player and the gcc build completely jam.

Quoted 7 lines

Quoted 4 lines
Quoted 1 line
Which sound system, and which sound card driver?
The driver is for the UDA1341 on a SA1110 chip written by myself. It is fully OSS compliant, no ALSA.
Your system should be able to handle that easily. Do you have some meminfo output to look at? What about 2.4.9?

OK now I have some comparative data. Two kernels:

1) based on 2.4.8-ac9 + Riel's VM fixes from -ac10
2) based on 2.4.9 with no extra VM patches

The test:
=========

This board has 32MB RAM, no swap. Root fs is NFS. On the serial console I start a command line mp3 player. In a first telnet session I start a build of gcc 3.0 (./configure; make). In a second telnet session I start 'top'. Music plays while gcc builds and I can see the CPU usage within top. Pretty real scenario, real life situation, expected system load, no trick.

2.4.8-ac9 plus -ac10 VM fixes:
==============================

Everything looks fine for a while. But all of a sudden: no more music, the gcc build stalls, system looks dead. In my first description of the problem quoted above I said that the audio was spitting glitches of 100 ms every 10 secs or so. This time there is nothing. The only difference is the addition of 'top' in the process list. Even the telnet sessions aren't echoing keystrokes anymore. Only the serial console echoes characters so kernel BH's are still running. Sysrq on the serial console works fortunately. kswapd is looping like crazy:

kswapd R C0023CC0 0 4 0 5 3 (L-TLB)

At first NFS traffic occupied the full network bandwidth to page stuff in. But after a while the following was printed to the console:

nfs: task 41867 can't get a request slot
nfs: task 41868 can't get a request slot
nfs: task 41869 can't get a request slot

and then no more NFS packet on the network. It is possible to ping the box which means that the net BH still works.

A couple sysrq-P at random intervals shows the CPU looping in the following functions:

PC value System.map
-------- ----------
c0040d84 zone_inactive_plenty
c0041024 try_to_swap_out
c00216e0 cpu_sa1100_cache_clean_invalidate_range
c00216d0 cpu_sa1100_cache_clean_invalidate_range
c0041304 swap_out_mm
c0041168 swap_out_pmd
c0044324 __get_swap_page
c0040d60 zone_inactive_plenty
c0041128 swap_out_pmd
c0040fec try_to_swap_out

Sysrq-M gives:

SysRq : Show Memory
Mem-info:
Free pages: 1012kB ( 0kB HighMem)
( Active: 2007, inactive_dirty: 0, inactive_clean: 0, free: 253 (255 510 765) ) 3*4kB 1*8kB 0*16kB 1*32kB 1*64kB 1*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB = 1012kB) = 0kB)
= 0kB)
Swap cache: add 0, delete 0, find 0/0
Free swap: 0kB
8192 pages of RAM
393 free pages
636 reserved pages
33 pages shared
0 pages swap cached
6 page tables cached
Buffer memory: 8000kB

2.4.9 kernel:
=============

Unlike the first (quoted) run, this kernel completely stalled when the jam conditions were reached just like the run described above. I mean here there is no audio stuttering at all, no echo from telnet sessions, nothing in user space gets to run anymore. kswapd is well locked in the R state:

kswapd R C0022CA0 0 4 0 5 3 (L-TLB)

Kernel interrupts and BHs still work i.e. I can ping the box, the serial console still echoes characters (kernel termios), and sysrq works but that's all. What's also interesting here is the fact that there is absolutely no NFS traffic going on.

A couple sysrq-P at random intervals shows the CPU looping in the following functions:

PC value System.map
-------- ----------
c003f6e0 zone_inactive_plenty
c003fa58 swap_out_pmd
c0060324 prune_icache
c003f6fc zone_inactive_plenty
c003faac swap_out_pmd
c00209a0 cpu_sa1100_set_pte
c003fc80 swap_out_mm
c0040c10 refill_inactive_scan
c00206c0 cpu_sa1100_cache_clean_invalidate_range
c003fa90 swap_out_pmd

Sysrq-M gives:

SysRq: Show Memory
Mem-info:
Free pages: 1008kB ( 0kB HighMem)
( Active: 2546, inactive_dirty: 63, inactive_clean: 0, free: 252 (255 510 765) ) 0*4kB 0*8kB 1*16kB 1*32kB 1*64kB 1*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB = 1008kB) = 0kB)
= 0kB)
Swap cache: add 0, delete 0, find 0/0
Free swap: 0kB
8192 pages of RAM
392 free pages
602 reserved pages
576 pages shared
0 pages swap cached
8 page tables cached
Buffer memory: 8000kB

Notes:
======

- Both kernels go flat after approx 14 min of mp3 playback.
- With enough retries (i.e. reboots) the gcc build succeed, so there is no lack of ressources in theory.
- SEnding a signal to the mp3 player (^C on the serial console) doesn't change anything.
- The only way out is the reset button.
- Both kernels (with or without -ac) deadlock.
- The same behavior occurs with 2.4.8-ac4.

Hope this helps somehow. If I can provide anything else please ask.

Nicolas

Steve Kieu

2001-08-25 04:31 UTC #10 lore raw

--- Nicolas Pitre <[email protected]> wrote: >

Quoted 43 lines · Daniel Phillips

Quoted 5 lines · Nicolas Pitre
Quoted 3 lines · Daniel Phillips
Quoted 2 lines
On August 24, 2001 08:14 pm, Nicolas Pitre
Quoted 1 line · wrote:
Quoted 1 line
Quoted 1 line
Quoted 1 line
I have a totally different setup but I can
reproduce the same behavior on
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
the system I have here:
ARM board with 32 MB RAM, no cache, NFS
root.
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
The kernel is based on 2.4.8-ac9 plus some
small VM fixes from -ac10.
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
My test consist in compiling gcc 3.0 while
some MP3s are continously playing
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
in the background. The gcc build goes
pretty far along until both the mp3
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
player and the gcc build completely jam.
Which sound system, and which sound card
driver?
Quoted 2 lines
Quoted 2 lines
The driver is for the UDA1341 on a SA1110 chip
written by myself. It is
Quoted 3 lines
Quoted 1 line
fully OSS compliant, no ALSA.
Your system should be able to handle that easily.
Do you have some meminfo
Quoted 1 line
output to look at? What about 2.4.9?
OK now I have some comparative data. Two kernels:
1) based on 2.4.8-ac9 + Riel's VM fixes from -ac10
2) based on 2.4.9 with no extra VM patches
The test:
=========

Quoted 10 lines

This board has 32MB RAM, no swap. Root fs is NFS. On the serial console I
start a command line mp3 player. In a first telnet session I start a build
of gcc 3.0 (./configure; make). In a second telnet session I start 'top'.
Music plays while gcc builds and I can see the CPU usage within top.
Pretty real scenario, real life situation, expected system load, no trick.

Strange enough to me, I can not reproduce this with my computer nearly the same as your described situation

Except nfs root file system, I started several rxvt play mpg123 ; compile the kernel ; browsing the net using mozilla, running top at one rxvt, kernel uses 2.4.9 is fine; 2.4.6 is fine too.

My computer is 686 128Mb RAM only 72Mb swap

OOP, 2.4.9 applied the one small patch for VM fix (just one line

Quoted 140 lines

2.4.8-ac9 plus -ac10 VM fixes:
==============================
Everything looks fine for a while. But all of a
sudden: no more music, the
gcc build stalls, system looks dead. In my first description of the problem
quoted above I said that the audio was spitting glitches of 100 ms every 10
secs or so. This time there is nothing. The only difference is the
addition of 'top' in the process list. Even the telnet sessions aren't
echoing keystrokes anymore. Only the serial console echoes characters so
kernel BH's are still running. Sysrq on the serial console works
fortunately. kswapd is looping like crazy:
kswapd R C0023CC0 0 4 0
5 3 (L-TLB)
At first NFS traffic occupied the full network bandwidth to page stuff in.
But after a while the following was printed to the console:
nfs: task 41867 can't get a request slot
nfs: task 41868 can't get a request slot
nfs: task 41869 can't get a request slot
and then no more NFS packet on the network. It is possible to ping the box
which means that the net BH still works.
A couple sysrq-P at random intervals shows the CPU looping in the following
functions:
PC value System.map
-------- ----------
c0040d84 zone_inactive_plenty
c0041024 try_to_swap_out
c00216e0 cpu_sa1100_cache_clean_invalidate_range
c00216d0 cpu_sa1100_cache_clean_invalidate_range
c0041304 swap_out_mm
c0041168 swap_out_pmd
c0044324 __get_swap_page
c0040d60 zone_inactive_plenty
c0041128 swap_out_pmd
c0040fec try_to_swap_out
Sysrq-M gives:
SysRq : Show Memory
Mem-info:
Free pages: 1012kB ( 0kB HighMem)
( Active: 2007, inactive_dirty: 0, inactive_clean:
0, free: 253 (255 510 765) )
3*4kB 1*8kB 0*16kB 1*32kB 1*64kB 1*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB = 1012kB)
= 0kB)
= 0kB)
Swap cache: add 0, delete 0, find 0/0
Free swap: 0kB
8192 pages of RAM
393 free pages
636 reserved pages
33 pages shared
0 pages swap cached
6 page tables cached
Buffer memory: 8000kB
2.4.9 kernel:
=============
Unlike the first (quoted) run, this kernel completely stalled when the jam
conditions were reached just like the run described above. I mean here
there is no audio stuttering at all, no echo from telnet sessions, nothing
in user space gets to run anymore. kswapd is well locked in the R state:
kswapd R C0022CA0 0 4 0
5 3 (L-TLB)
Kernel interrupts and BHs still work i.e. I can ping the box, the serial
console still echoes characters (kernel termios), and sysrq works but that's
all. What's also interesting here is the fact that there is absolutely no
NFS traffic going on.
A couple sysrq-P at random intervals shows the CPU looping in the following
functions:
PC value System.map
-------- ----------
c003f6e0 zone_inactive_plenty
c003fa58 swap_out_pmd
c0060324 prune_icache
c003f6fc zone_inactive_plenty
c003faac swap_out_pmd
c00209a0 cpu_sa1100_set_pte
c003fc80 swap_out_mm
c0040c10 refill_inactive_scan
c00206c0 cpu_sa1100_cache_clean_invalidate_range
c003fa90 swap_out_pmd
Sysrq-M gives:
SysRq: Show Memory
Mem-info:
Free pages: 1008kB ( 0kB HighMem)
( Active: 2546, inactive_dirty: 63, inactive_clean:
0, free: 252 (255 510 765) )
0*4kB 0*8kB 1*16kB 1*32kB 1*64kB 1*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB = 1008kB)
= 0kB)
= 0kB)
Swap cache: add 0, delete 0, find 0/0
Free swap: 0kB
8192 pages of RAM
392 free pages
602 reserved pages
576 pages shared
0 pages swap cached
8 page tables cached
Buffer memory: 8000kB
Notes:
======

Quoted 7 lines

- Both kernels go flat after approx 14 min of mp3 playback.
- With enough retries (i.e. reboots) the gcc build succeed, so there is no
lack of ressources in theory.
- SEnding a signal to the mp3 player (^C on the serial

=== message truncated ===

=====
S.KIEU

_____________________________________________________________________________
http://shopping.yahoo.com.au - Father's Day Shopping
- Find the perfect gift for your Dad for Father's Day

Daniel Phillips

2001-08-25 16:31 UTC #11 lore raw

In reply to Nicolas Pitre

Quoted 5 lines · Nicolas Pitre

This board has 32MB RAM, no swap. Root fs is NFS. On the serial console I start a command line mp3 player. In a first telnet session I start a build of gcc 3.0 (./configure; make). In a second telnet session I start 'top'. Music plays while gcc builds and I can see the CPU usage within top. Pretty real scenario, real life situation, expected system load, no trick.

You're streaming the mp3 over nfs, right? From your setup I'd guess there's no local hard disk.

Quoted 3 lines

nfs: task 41867 can't get a request slot
nfs: task 41868 can't get a request slot
nfs: task 41869 can't get a request slot

Uhuh. Would you please look in your logs for "allocation failed" messages? (Side note: reading the nfs code now... to whose eyes are names like tk_rqstp beautiful?)

Quoted 1 line

( Active: 2007, inactive_dirty: 0, inactive_clean: 0, free: 253 (255 510

765) )

Whoops, nothing inactive but kswap going full blast. We're getting warmer.

Quoted 7 lines

2.4.9 kernel:
=============
Unlike the first (quoted) run, this kernel completely stalled when the jam conditions were reached just like the run described above. I mean here there is no audio stuttering at all, no echo from telnet sessions, nothing in user space gets to run anymore.

Yes, a slight difference, however they are both wedged in the same way, from your task samples:

Quoted 13 lines

PC value System.map
-------- ----------
c003f6e0 zone_inactive_plenty
c003fa58 swap_out_pmd
c0060324 prune_icache
c003f6fc zone_inactive_plenty
c003faac swap_out_pmd
c00209a0 cpu_sa1100_set_pte
c003fc80 swap_out_mm
c0040c10 refill_inactive_scan
c00206c0 cpu_sa1100_cache_clean_invalidate_range
c003fa90 swap_out_pmd

Quoted 4 lines

Kernel interrupts and BHs still work i.e. I can ping the box, the serial console still echoes characters (kernel termios), and sysrq works but that's all. What's also interesting here is the fact that there is absolutely no NFS traffic going on.

That's understandable. Everything that needs to allocate memory is wedged.

Quoted 1 line

- The same behavior occurs with 2.4.8-ac4.

How far back do you have to go before you get one that works? I seem to recall the inactive_plenty changes came in at 2.4.8-pre1. Could you try it with 2.4.7, please.

--
Daniel

Nicolas Pitre

2001-08-25 18:29 UTC #12 lore raw

Quoted 9 lines · Daniel Phillips

Quoted 5 lines · Nicolas Pitre
This board has 32MB RAM, no swap. Root fs is NFS. On the serial console I start a command line mp3 player. In a first telnet session I start a build of gcc 3.0 (./configure; make). In a second telnet session I start 'top'. Music plays while gcc builds and I can see the CPU usage within top. Pretty real scenario, real life situation, expected system load, no trick.
You're streaming the mp3 over nfs, right? From your setup I'd guess there's no local hard disk.

No there isn't.

Quoted 5 lines

Quoted 1 line
- The same behavior occurs with 2.4.8-ac4.
How far back do you have to go before you get one that works? I seem to recall the inactive_plenty changes came in at 2.4.8-pre1. Could you try it with 2.4.7, please.

2.4.7 does the same, spinning in kswapd while everything else is stalled, except for kernel BHs.

SysRq: Show Memory
Mem-info:
Free pages: 1016kB ( 0kB HighMem)
( Active: 2554, inactive_dirty: 0, inactive_clean: 0, free: 254 (255 510 765) ) 4*4kB 1*8kB 0*16kB 1*32kB 1*64kB 1*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB = 1016kB) = 0kB)
= 0kB)
Swap cache: add 0, delete 0, find 0/0
Free swap: 0kB
8192 pages of RAM
395 free pages
626 reserved pages
581 pages shared
0 pages swap cached
3 page tables cached
Buffer memory: 8000kB

So 2.4.7 was screwed the same way too.

Nicolas

Russell King

2001-08-25 18:51 UTC #13 lore raw

In reply to Daniel Phillips

Ok, I'm saying this for the record, Nico already knows the following ;)

Quoted 8 lines · Nicolas Pitre

SysRq: Show Memory
Mem-info:
Free pages: 1016kB ( 0kB HighMem)
( Active: 2554, inactive_dirty: 0, inactive_clean: 0, free: 254 (255 510 765) ) 4*4kB 1*8kB 0*16kB 1*32kB 1*64kB 1*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB = 1016kB) = 0kB)
= 0kB)
Swap cache: add 0, delete 0, find 0/0

Quoted 8 lines

Free swap: 0kB
8192 pages of RAM
395 free pages
626 reserved pages
581 pages shared
0 pages swap cached
3 page tables cached
Buffer memory: 8000kB

The above buffer memory usage is caused by an 8MB ramdisk. Unfortunately, we are looking by default for 8192 * (2 + 2) / 100 = 327 pages between the buffermem and the page cache before triggering the OOM handler.

With an 8MB ramdisk (== 2000 pages) obviously we'll never reach that, so we might as well not have the oom handler in this case... or...

With the attached patch, we take the number of ramdisk pages into account when checking for OOM. (side note: we don't seem to account for ramdisk pages, therefore I have to count the individual pages on the active list).

The patch below factors out the fixed ramdisk allocation, and allows the OOM killer to be functional on machines with ramdisks. We only count the number of ramdisk pages when we're getting close to the limit (ie, freepages stuff indicates oom, and there's no swap).

As an added bonus, this patch also dumps out the number of Page Cache pages, buffermem pages, and ramdisk pages on sysrq-m.

Note that this doesn't solve Nico's original problem.

mm/oom_kill.c +7 -4

diff -x .* -x *.[oas] -urN ref/mm/oom_kill.c linux/mm/oom_kill.c
@@ -193,6 +193,8 @@
 	return;
 }
 
+extern long count_ramdisk_pages(void);
+
 /**
  * out_of_memory - is the system out of memory?
  *

@@ -210,6 +212,10 @@
 	if (nr_free_pages() + nr_inactive_clean_pages() > freepages.low)
 		return 0;
 
+	/* Enough swap space left?  Not OOM. */
+	if (nr_swap_pages > 0)
+		return 0;
+
 	/*
 	 * If the buffer and page cache (excluding swap cache) are over
 	 * their (/proc tunable) minimum, we're still not OOM.  We test

@@ -219,14 +225,11 @@
 	cache_mem = atomic_read(&page_cache_size);
 	cache_mem += atomic_read(&buffermem_pages);
 	cache_mem -= swapper_space.nrpages;
+	cache_mem -= count_ramdisk_pages();
 	limit = (page_cache.min_percent + buffer_mem.min_percent);
 	limit *= num_physpages / 100;
 
 	if (cache_mem > limit)
-		return 0;
-
-	/* Enough swap space left?  Not OOM. */
-	if (nr_swap_pages > 0)
 		return 0;
 
 	/* Else... */

mm/page_alloc.c +6 -0

diff -x .* -x *.[oas] -urN ref/mm/page_alloc.c linux/mm/page_alloc.c
@@ -690,6 +690,8 @@
      return (sum > 0 ? sum : 0);
 }
 
+extern long count_ramdisk_pages(void);
+
 /*
  * Show free area list (used inside shift_scroll-lock stuff)
  * We also calculate the percentage fragmentation. We do this by counting the

@@ -743,6 +745,10 @@
 #ifdef SWAP_CACHE_INFO
 	show_swap_cache_info();
 #endif	
+
+	printk("Page cache size: %d\n", atomic_read(&page_cache_size));
+	printk("Buffer mem: %d\n", atomic_read(&buffermem_pages));
+	printk("Ramdisk pages: %ld\n", count_ramdisk_pages());
 }
 
 void show_free_areas(void)

mm/vmscan.c +18 -0

diff -x .* -x *.[oas] -urN ref/mm/vmscan.c linux/mm/vmscan.c
@@ -816,6 +816,24 @@
 	return nr_deactivated;
 }
 
+long count_ramdisk_pages(void)
+{
+	struct list_head *page_lru;
+	struct page *page;
+	long nr_ramdisk = 0;
+
+	spin_lock(&pagemap_lru_lock);
+	for (page_lru = active_list.next; page_lru != &active_list;
+	     page_lru = page_lru->next) {
+		page = list_entry(page_lru, struct page, lru);
+		if (page_ramdisk(page))
+			nr_ramdisk ++;
+	}
+	spin_unlock(&pagemap_lru_lock);
+
+	return nr_ramdisk;
+}
+
 /*
  * Check if there are zones with a severe shortage of free pages,
  * or if all zones have a minor shortage.

--
Russell King ([email protected]) The developer of ARM Linux
http://www.arm.linux.org.uk/personal/aboutme.html

Russell King

2001-08-25 08:10 UTC #14 lore raw

In reply to Daniel Phillips

Quoted 1 line · Nicolas Pitre

6 page tables cached

Although this won't help the basic problem, there could me as much as 100K cached in those page tables. I wonder if we could hook the pgt cache into the VM cache shrinking, so we can free most, if not all of this cache (rather than it being in the idle loop only).

I'll look into it, produce a patch, but I'm not a VM hacker.

--
Russell King ([email protected]) The developer of ARM Linux
http://www.arm.linux.org.uk/personal/aboutme.html

Daniel Phillips

2001-08-25 16:31 UTC #15 lore raw

In reply to Daniel Phillips

Quoted 9 lines · Russell King

Quoted 1 line · Nicolas Pitre
6 page tables cached
Although this won't help the basic problem, there could me as much as 100K cached in those page tables. I wonder if we could hook the pgt cache into the VM cache shrinking, so we can free most, if not all of this cache (rather than it being in the idle loop only).
I'll look into it, produce a patch, but I'm not a VM hacker.

You know what a pte is so you're a VM hacker ;-)

--
Daniel