Skip to contentSkip to conversation

Kernel doesn't free Cached Memory

DISCUSSION Vinicius 6 participants 7 messages Jul 22, 2005 – Jul 26, 2005

lore
Share & tools
lore

Vinicius

author
#0 lore raw

Hi all!

I have a server with 2 Pentium 4 HT processors and 32 GB of RAM, this
server runs lots of applications that consume lots of memory to. When I stop this applications, the kernel doesn't free memory (the memory still in use) and the server cache lots of memory (~27GB). When I start this applications, the kernel sends "Out of Memory" messages and kill some random applications.

Anyone know how can I reduce the kernel cached memory on RHEL 3 (kernel
2.4.21-32.ELsmp - Trial version)? There is a way to reduce the kernel cached memory utilization?

Thanks in advance (sorry my bad english).

Vinicius.
Protolink Consultoria.

#1 lore raw

On Fri, 2005-07-22 at 08:27 -0300, Vinicius wrote:
[...]

Quoted 10 lines

I have a server with 2 Pentium 4 HT processors and 32 GB of RAM, this
server runs lots of applications that consume lots of memory to. When I stop this applications, the kernel doesn't free memory (the memory still in use) and the server cache lots of memory (~27GB). When I start this applications, the kernel sends "Out of Memory" messages and kill some random applications.

Anyone know how can I reduce the kernel cached memory on RHEL 3 (kernel
2.4.21-32.ELsmp - Trial version)? There is a way to reduce the kernel cached memory utilization?

Probably RedHat's support can answer this for RHEL 3.

Bernd

Signature · 3 lines
Firmix Software GmbH                   http://www.firmix.at/
mobil: +43 664 4416156                 fax: +43 1 7890849-55
          Embedded Linux Development and Services
#2 lore raw
Quoted 13 lines · Bernd Petrovitsch

On Fri, 2005-07-22 at 08:27 -0300, Vinicius wrote:
[...]

Quoted 10 lines

I have a server with 2 Pentium 4 HT processors and 32 GB of RAM, this
server runs lots of applications that consume lots of memory to. When I stop this applications, the kernel doesn't free memory (the memory still in use) and the server cache lots of memory (~27GB). When I start this applications, the kernel sends "Out of Memory" messages and kill some random applications.

Anyone know how can I reduce the kernel cached memory on RHEL 3 (kernel
2.4.21-32.ELsmp - Trial version)? There is a way to reduce the kernel cached memory utilization?

Quoted 8 lines

Probably RedHat's support can answer this for RHEL 3.

Bernd
--
Firmix Software GmbH http://www.firmix.at/
mobil: +43 664 4416156 fax: +43 1 7890849-55
Embedded Linux Development and Services

How do you know the memory is "still in use?" I think that if you run a program that uses so much memory that some swap is used, but not allowing all free memory to be exhausted, you can run the same program over and over again forever. Therefore, the memory must have been freed and available for the next time you run the program.

The problem is that you are exhausting all free memory causing the kernel to kill processes just to stay alive. Then you look at some memory "tool" and think that it shows something about what is, or has been freed. You need to set process limits on the amount of resources each process can use and, depending upon the implementation details of your "server", you might even need to turn memory over-commit off:
echo "0" > /proc/sys/vm/overcommit_memory

The fact is that unused memory is stored in caches and buffers. That's how it doesn't get wasted. In fact, with a busy system an attempt is made to use all available real RAM and minimize swap.

Some server programs are written much like your defective test program. You can't attempt to allocate all virtual memory using malloc(), expecting it to return NULL when there is no more. Malloc() will return NULL when it has gotten trashed by somebody overwriting buffer boundaries or when address-space is exhausted.

It has no clue how much RAM is available and, in fact if it did, it would represent an information leak. Programs should allocate what they need, not what they can get.

Cheers,
Dick Johnson
Penguin : Linux version 2.6.12 on an i686 machine (5537.79 BogoMips). Warning : 98.36% of all statistics are fiction.
.

****************************************************************
The information transmitted in this message is confidential and may be privileged. Any review, retransmission, dissemination, or other use of this information by persons or entities other than the intended recipient is prohibited. If you are not the intended recipient, please notify Analogic Corporation immediately - by replying to this message or by sending an email to [email protected] - and destroy all copies of this information, including any attachments, without reading or disclosing them.

Thank you.

#3 lore raw

Dick Johnson wrote: {

Quoted 8 lines · Vinicius

[...]

Quoted 6 lines

I have a server with 2 Pentium 4 HT processors and 32 GB of RAM,
this server runs lots of applications that consume lots of memory to. When I stop this applications, the kernel doesn't free memory (the memory still in use) and the server cache lots of memory (~27GB). When I start this applications, the kernel sends "Out of Memory" messages and kill some random applications.

...you might even need to turn memory over-commit off:
echo "0" > /proc/sys/vm/overcommit_memory
}

That's in 2.4. In 2.6 it's:
echo "2" > /proc/sys/vm/overcommit_memory

But the kernel doesn't honor no-overcommit in either version, i.e. it still overcommits/pages-out loaded/running procs, thus invoking OOM!

Is there a way to make the kernel strictly honor the no-overcommit request?

Thanks!

#4 lore raw
Quoted 9 lines · Al Boldi

Dick Johnson wrote: {

Quoted 5 lines · Vinicius

[...]

Quoted 2 lines

I have a server with 2 Pentium 4 HT processors and 32 GB of RAM,
this server runs lots of applications that consume lots of memory to. When I stop this applications, the kernel doesn't free memory (the memory still in use) and the server cache lots of memory (~27GB). When I start this applications, the kernel sends "Out of Memory" messages and kill some random applications.

Quoted 6 lines

...you might even need to turn memory over-commit off:
echo "0" > /proc/sys/vm/overcommit_memory
}

That's in 2.4. In 2.6 it's:
echo "2" > /proc/sys/vm/overcommit_memory

RHEL3 *is* a 2.4 kernel.

Quoted 6 lines

But the kernel doesn't honor no-overcommit in either version, i.e. it still overcommits/pages-out loaded/running procs, thus invoking OOM!

Is there a way to make the kernel strictly honor the no-overcommit request?

Don't have swap?

Signature · 3 lines
   -bill davidsen ([email protected])
"The secret to procrastination is to put things off until the
 last possible moment - but no longer"  -me
#5 lore raw

Bill Davidsen wrote: {

Quoted 13 lines · Al Boldi

Dick Johnson wrote: {

Quoted 9 lines · Vinicius

[...]

Quoted 6 lines

I have a server with 2 Pentium 4 HT processors and 32 GB of RAM,
this server runs lots of applications that consume lots of memory to. When I stop this applications, the kernel doesn't free memory (the memory still in use) and the server cache lots of memory (~27GB). When I start this applications, the kernel sends "Out of Memory" messages and kill some random applications.

Quoted 6 lines

...you might even need to turn memory over-commit off:
echo "0" > /proc/sys/vm/overcommit_memory
}

That's in 2.4. In 2.6 it's:
echo "2" > /proc/sys/vm/overcommit_memory

RHEL3 *is* a 2.4 kernel.

Quoted 2 lines

But the kernel doesn't honor no-overcommit in either version, i.e. it

still

Quoted 3 lines

overcommits/pages-out loaded/running procs, thus invoking OOM!

Is there a way to make the kernel strictly honor the no-overcommit

request?

Quoted 1 line

Don't have swap?
}

Turn off swap and things get worse!

Paolo Ornati wrote:{

Quoted 6 lines · Bill Davidsen

And IMHO Linux is *way* too willing to evicy clean pages of my programs to use as disk buffer, so that when system memory is full I pay the overhead of TWO disk i/o's, one to finally write the data to the disk and one to read my program back in. If free software is about choice, I wish there was more in the area of how memory is used.

isn't this tuned enough by "/proc/sys/vm/swappiness" ? }

Swappiness tunes but does not inhibit overcommit!

So the question remains:
Why Is there no way to make the kernel _strictly_ honor the
no-overcommit request?

--
Al

#6 lore raw
In reply to Vinicius
Quoted 5 lines · Vinicius

Hi all!

I have a server with 2 Pentium 4 HT processors and 32 GB of RAM, this
server runs lots of applications that consume lots of memory to. When I stop this applications, the kernel doesn't free memory (the memory still in use)

See any FAQ on the Linux memory management - memory is reclaimed when needed not when nobody is using it. That makes things more efficient.

Quoted 3 lines

and the server cache lots of memory (~27GB). When I start this applications, the kernel sends "Out of Memory" messages and kill some random applications.

Some RHEL3 kernels had a problem with very large memory sizes and 2.4. That should not be the case in the current RHEL3 kernels. 2.6 handles very large systems a lot lot better, and of course the fact real computers now have 64bit processors has also rather improved life.

Alan