Skip to contentSkip to conversation

Kernel cached memory

DISCUSSION Ashley 10 participants 14 messages Jul 22, 2005 – Aug 1, 2005

lore
Share & tools
lore

Ashley

author
#0 lore raw

Hi all
I've a server with 2 Operton 64bit CPU and 12G memory, and this server
is used to run applications which will comsume huge memory,
the problem is: when this aplications exits, the free memory of the server is still very low(accroding to the output of "top"), and from the output of command "free", I can see that many GB memory was cached by kernel. Does anyone know how to free the kernel cached memory? thanks in advance.

Sincerely,
Ashley
------------------------------------------------------
AlChip Technologies, Limited
26F, Huamin Empire Plaza, Suite B-H,
726 West YanAn Road,
Shanghai, China 200050
Tel: 86-21-5237-8800 ext.220
Fax: 86-21-5237-8880
http://www.alchip.com
------------------------------------------------------

#1 lore raw
Quoted 7 lines · Ashley

I've a server with 2 Operton 64bit CPU and 12G memory, and this server
is used to run applications which will comsume huge memory, the problem is: when this aplications exits, the free memory of the server is still very low(accroding to the output of "top"), and
from the output of command "free", I can see that many GB memory was cached by kernel. Does anyone know how to free the kernel cached memory? thanks in advance.

Free memory is bad, it means the memory doesn't have a proper use. Cached memory will be freed automatically when the kernel needs memory for other (more important) things.

Erik

Signature · 2 lines
+-- Erik Mouw -- www.harddisk-recovery.com -- +31 70 370 12 90 --
| Lab address: Delftechpark 26, 2628 XH, Delft, The Netherlands
#4 lore raw
In reply to Ashley
Quoted 3 lines · Ashley

from the output of command "free", I can see that many GB memory was cached by kernel. Does anyone know how to free the kernel cached memory? thanks in advance.

You don't want that. Kernel will free cached memory when apps need it. When there's a lot of free memory, linux always tries to use it for cache - because caches speed up things, and it's silly to to leave memory free if you can use it for caching.

#5 lore raw
In reply to Ashley
Quoted 7 lines · Ashley

I've a server with 2 Operton 64bit CPU and 12G memory, and this server
is used to run applications which will comsume huge memory, the problem is: when this aplications exits, the free memory of the server is still very low(accroding to the output of "top"), and
from the output of command "free", I can see that many GB memory was cached by kernel. Does anyone know how to free the kernel cached memory? thanks in advance.

It's a very - very - very old and bad logic (at least nowdays) from the stone age to free up memory. Memory is for using ... If you have memory why don't you want to use? For an actively used system, memory should be used as much as possible to maximize the performance. The only problem when kernel does not want to "give back" used memory for eg caching for an application though but it's another problem ...

Anyway, want to have 'free memory' is a thing like having dozens of cars in your garage which don't want to be used ...

Signature · 1 line
- Gábor
#6 lore raw
Quoted 3 lines · Gábor Lénárt

Anyway, want to have 'free memory' is a thing like having dozens of cars in your garage which don't want to be used ...

Really? I thought it was good to leave some memory free to speed up application startup, so we don't have to evict a bunch of pages first.

Lee

#2 lore raw
In reply to Erik Mouw

Wouldn't having (practically) all your memory used for cache slow down starting a new program? First it would have to free up that space, and then put stuff in that space, taking potentially twice as long. I think there should be a system call for freeing cached memory, for those that do want to do it.

-----Original Message-----
From: [email protected] [mailto:[email protected]]On Behalf Of Erik Mouw
Sent: Friday, July 22, 2005 5:58 AM
To: Ashley
Cc: [email protected]
Subject: Re: Kernel cached memory

Quoted 3 lines · Ashley

I've a server with 2 Operton 64bit CPU and 12G memory, and this server
is used to run applications which will comsume huge memory, the problem is: when this aplications exits, the free memory of the

server

Quoted 2 lines

is still very low(accroding to the output of "top"), and from the output of command "free", I can see that many GB memory was

cached

Quoted 2 lines

by kernel. Does anyone know how to free the kernel cached memory? thanks in advance.

Free memory is bad, it means the memory doesn't have a proper use. Cached memory will be freed automatically when the kernel needs memory for other (more important) things.

Erik

--

+-- Erik Mouw -- www.harddisk-recovery.com -- +31 70 370 12 90 --
| Lab address: Delftechpark 26, 2628 XH, Delft, The Netherlands

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

#7 lore raw
In reply to Lee Revell

Am Freitag, 22. Juli 2005 19:58 schrieb Lee Revell:

Quoted 7 lines · Gábor Lénárt
Quoted 3 lines

Anyway, want to have 'free memory' is a thing like having dozens of cars in your garage which don't want to be used ...

Really? I thought it was good to leave some memory free to speed up application startup, so we don't have to evict a bunch of pages first.

Eviction is cheap when the pages are clean.

Regards
Oliver

#3 lore raw
In reply to John Pearson
Quoted 5 lines · John Pearson

Wouldn't having (practically) all your memory used for cache slow down starting a new program? First it would have to free up that space, and then put stuff in that space, taking potentially twice as long. I think there should be a system call for freeing cached memory, for those that do want to do it.

I think this one is good enough:

#include <stdlib.h>

int main() {
    void *p;
    unsigned size = 1<<20;
    unsigned long total=0;
    while(size) {
        p = malloc(size);
        if(!p) size>>=1;
        else {
            memset(p, 0x77, size);
            total+=size;
            printf("Allocated %9u bytes, %12lu total\n",size,total);
        }
    }
    return 0;
}

You may want to adapt it so that it takes an argument now many megabytes to eat before it dies.
--
vda

#8 lore raw
In reply to Gábor Lénárt
Quoted 11 lines · Gábor Lénárt
Quoted 5 lines · Ashley

I've a server with 2 Operton 64bit CPU and 12G memory, and this server
is used to run applications which will comsume huge memory,
the problem is: when this aplications exits, the free memory of the server is still very low(accroding to the output of "top"), and from the output of command "free", I can see that many GB memory was cached by kernel. Does anyone know how to free the kernel cached memory? thanks in advance.

It's a very - very - very old and bad logic (at least nowdays) from the stone age to free up memory.

It's very Microsoft to claim that the OS always knows best, and not let the user tune the system the way they want it tuned. And if that means to leave a bunch of free memory for absolute fastest availability, the admin should have that option.

Quoted 5 lines

Memory is for using ... If you have memory why
don't you want to use? For an actively used system, memory should be used as much as possible to maximize the performance. The only problem when kernel does not want to "give back" used memory for eg caching for an application though but it's another problem ...

No, that's the problem he's trying to address.

Quoted 4 lines

Anyway, want to have 'free memory' is a thing like having dozens of cars in your garage which don't want to be used ...

Which wait to be used when needed, rather than after someone cleans a bunch of old junk out of them and scurries around the garage looking for the right car to let you use.

Just because default operation works well for you, kindly don't try to convice us that there are no cases when the default operation is NOT optimal. And IMHO Linux is *way* too willing to evicy clean pages of my programs to use as disk buffer, so that when system memory is full I pay the overhead of TWO disk i/o's, one to finally write the data to the disk and one to read my program back in. If free software is about choice, I wish there was more in the area of how memory is used.

Signature · 3 lines
   -bill davidsen ([email protected])
"The secret to procrastination is to put things off until the
 last possible moment - but no longer"  -me
#9 lore raw
Quoted 7 lines · Bill Davidsen

Just because default operation works well for you, kindly don't try to convice us that there are no cases when the default operation is NOT optimal. And IMHO Linux is *way* too willing to evicy clean pages of my programs to use as disk buffer, so that when system memory is full I pay the overhead of TWO disk i/o's, one to finally write the data to the disk and one to read my program back in. If free software is about choice, I wish there was more in the area of how memory is used.

You must be more specific here; cached memory isn't always dirty. Most of the times the pages used for cache are clean and can be evicted quite fast and of course without writting anything to the disk.

If you want to tune the balance between "pages used for programs" vs "pages used for cache" that's another problem, but I can't find a reason why kernel shouldn't cache things when free memory is available. And certainly that's not what Ashley was asking.

#10 lore raw
In reply to Bill Davidsen
Quoted 6 lines · On Mon, 25 Jul 2005 12:47:50 -0400 Bill Davidsen <[email protected]> wrote:

And IMHO Linux is *way* too willing to evicy clean pages of my programs to use as disk buffer, so that when system memory is full I pay the overhead of TWO disk i/o's, one to finally write the data to the disk and one to read my program back in. If free software is about choice, I wish there was more in the area of how memory is used.

isn't this tuned enough by "/proc/sys/vm/swappiness" ?

Signature · 2 lines
	Paolo Ornati
	Linux 2.6.13-rc3 on x86_64
#11 lore raw
Quoted 13 lines · Paolo Ornati
Quoted 7 lines · On Mon, 25 Jul 2005 12:47:50 -0400 Bill Davidsen <[email protected]> wrote:

And IMHO Linux is *way* too willing to evicy clean pages of my programs to use as disk buffer, so that when system memory is full I pay the overhead of TWO disk i/o's, one to finally write the data to
the disk and one to read my program back in. If free software is
about choice, I wish there was more in the area of how memory is
used.

Quoted 4 lines

isn't this tuned enough by "/proc/sys/vm/swappiness" ?

Let me generate some data points for discussion. But the general answer is no, I just want to try to have some numbers to discuss. I also want to go back to some 2.4.xx-aa kernels, Andrea had some very nice things in his bdflush code, and he was kind enough to explain to me how to tune them so I avoided the worst case events.

Signature · 3 lines
bill davidsen <[email protected]>
 CTO TMR Associates, Inc
 Doing interesting things with small computers since 1979
#12 lore raw
In reply to Bill Davidsen
Quoted 12 lines · Bill Davidsen
Quoted 11 lines · Gábor Lénárt
Quoted 7 lines · Ashley

I've a server with 2 Operton 64bit CPU and 12G memory, and this server
is used to run applications which will comsume huge memory, the problem is: when this aplications exits, the free memory of the server is still very low(accroding to the output of "top"), and from the output of command "free", I can see that many GB memory was cached by kernel. Does anyone know how to free the kernel cached memory? thanks in advance.

Quoted 7 lines
Quoted 2 lines

It's a very - very - very old and bad logic (at least nowdays) from the stone age to free up memory.

It's very Microsoft to claim that the OS always knows best, and not let the user tune the system the way they want it tuned. And if that means to leave a bunch of free memory for absolute fastest availability, the admin should have that option.

Sure, sorry if my comment can be treated in this way ... I mean surprising amount of people I've met criticised Linux (well, some years ago when DOS was popular) that he/she want to see that 'free memory' field reported eg by 'top' should be the maximum all the time ... I mean this way: this is the behaviour which is quite wrong, I've written about this.

Sure, because of my not too good English, I may have missed the real meaning of the mail, sorry about it!

Signature · 1 line
- Gábor
#13 lore raw
Quoted 29 lines · Gábor Lénárt
Quoted 18 lines · Bill Davidsen
Quoted 15 lines · Gábor Lénárt
Quoted 11 lines · Ashley

I've a server with 2 Operton 64bit CPU and 12G memory, and this server is used to run applications which will comsume huge memory, the problem is: when this aplications exits, the free memory of the server is still very low(accroding to the output of "top"), and

from the output of command "free", I can see that many GB memory was

cached by kernel. Does anyone know how to free the kernel cached memory? thanks in advance.

Quoted 7 lines
Quoted 4 lines

It's a very - very - very old and bad logic (at least nowdays) from the stone age to free up memory.

It's very Microsoft to claim that the OS always knows best, and not let the user tune the system the way they want it tuned. And if that means to leave a bunch of free memory for absolute fastest availability, the admin should have that option.

Quoted 9 lines

Sure, sorry if my comment can be treated in this way ... I mean surprising amount of people I've met criticised Linux (well, some years ago when DOS was popular) that he/she want to see that 'free memory' field reported eg by 'top' should be the maximum all the time ... I mean this way: this is the behaviour which is quite wrong, I've written about this.

Sure, because of my not too good English, I may have missed the real meaning of the mail, sorry about it!

Well, I thought I understood "from the stone age" but I may have taken it slightly too literally. But I really would like to have more control over Linux memory use, because it does cause bad behaviour at times. If I have 4GB of RAM, I'd like to set 200MB or so aside for programs, and never page out the window I'm going to uncover later. Likewise when I write a DVD image, I would like to avoid buffering a few GB without i/o and then driving the disk totally busy while it gets written out (after it has pushed out things I will use again).

The old 2.4.x-aa kernels had some tunables to make the kernel aggressive about writing pages to disk quickly, and I haven't been able to match that behaviour without patches in 2.6. I may be missing a tunable, but swappiness doesn't seem to be the one I want. I have a patch I'm playing with, but it's not ready for prime time, and is probably counter to the current philosophy of memory management.

Thanks for clarifying.

Signature · 3 lines
bill davidsen <[email protected]>
 CTO TMR Associates, Inc
 Doing interesting things with small computers since 1979