buffer cache question
Hello, list,
I use 4 disk nodes with NBD.
All of my nodes have 2GB ram.
But the buffer cache newer rise over 830MB.
Is there some limit?
Where can i change this limit? (if it is)
Thanks,
Janos
[root@st-0001 root]# free
total used free shared buffers cached
Mem: 2073152 933188 1139964 0 836776 43416
-/+ buffers/cache: 52996 2020156
Swap: 0 0 0
[root@st-0001 root]# cat /proc/meminfo
MemTotal: 2073152 kB MemFree: 1139012 kB Buffers: 835928 kB Cached: 43448 kB SwapCached: 0 kB Active: 12872 kB Inactive: 871424 kB HighTotal: 1179584 kB HighFree: 1129764 kB LowTotal: 893568 kB LowFree: 9248 kB SwapTotal: 0 kB SwapFree: 0 kB Dirty: 0 kB Writeback: 0 kB Mapped: 9104 kB Slab: 30248 kB CommitLimit: 1036576 kB Committed_AS: 15428 kB PageTables: 408 kB VmallocTotal: 114680 kB VmallocUsed: 196 kB VmallocChunk: 114476 kB
[root@st-0001 root]#
Quoted 9 lines
-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of JaniD++
Sent: Sunday, December 18, 2005 3:09 PM
To: [email protected]
Subject: buffer cache question
Quoted 8 lines
Hello, list,
I use 4 disk nodes with NBD.
All of my nodes have 2GB ram.But the buffer cache newer rise over 830MB.
Is there some limit?
For writes only 40% of the ram can be "dirty".
Quoted 1 line
Where can i change this limit? (if it is)
I believe there is a way to change it, I am pretty sure that it has been discussed on the kernel list a couple of months ago, I don't remember exactly what the change is, and I think the change was more complicated that was obvious.
The previous subject on a similar thing was:
"kernel 2.6.13 buffer strangeness"
Roger
Quoted 44 lines
Thanks,
Janos[root@st-0001 root]# free
total used free shared
buffers cached
Mem: 2073152 933188 1139964 0
836776 43416
-/+ buffers/cache: 52996 2020156
Swap: 0 0 0
[root@st-0001 root]# cat/proc/meminfoMemTotal: 2073152 kB MemFree: 1139012 kB Buffers: 835928 kB Cached: 43448 kB SwapCached: 0 kB Active: 12872 kB Inactive: 871424 kB HighTotal: 1179584 kB HighFree: 1129764 kB LowTotal: 893568 kB LowFree: 9248 kB SwapTotal: 0 kB SwapFree: 0 kB Dirty: 0 kB Writeback: 0 kB Mapped: 9104 kB Slab: 30248 kB CommitLimit: 1036576 kB Committed_AS: 15428 kB PageTables: 408 kB VmallocTotal: 114680 kB VmallocUsed: 196 kB VmallocChunk: 114476 kB[root@st-0001 root]#
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected] More majordomo info at
http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Hi,
I have read back in the kernel-archives, and found this messages, about the same theme, but there is one difference!
On the old messages:
Quoted 8 lines
Nate Diller wrote:
just found the culprit. guess i should have read the code the first time.get_dirty_limits()indrivers/block/page_writeback.chas a hard-coded upper limit todirty_ratio. it's capped to half of the unmapped pages, so maybe 30-40% of your system's memory. so if you are brave, just remove the "/ 2" parts from the 'if (dirty_ratio>unmapped_ratio/ 2)dirty_ratio=unmapped_ratio/ 2;' check, and you can have all the OOM goodness you want.
...
Quoted 6 lines
I changed that bit of code to:
if (
dirty_ratio>unmapped_ratio- 10)
dirty_ratio=unmapped_ratio- 10;and added a couple of sanity checks so that it couldn't get below 5 or
above 95.
Quoted 2 lines
Then set
/proc/sys/vm/dirty_ratioto 95 anddirty_background_ratioto 1.
In this case, this modification is only for the *dirty* memory buffer. I want to use more buffer *cache*! :-)
The unwritten dirty memory ratio is good enough for me.
Anybody has an idea?
Cheers,
Janos
----- Original Message -----
From: "Roger Heflin" <[email protected]>
To: "'JaniD++'" <[email protected]>; <[email protected]>
Sent: Monday, December 19, 2005 7:15 PM
Subject: RE: buffer cache question
Quoted 82 lines
Quoted 9 lines
-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of JaniD++
Sent: Sunday, December 18, 2005 3:09 PM
To: [email protected]
Subject: buffer cache questionQuoted 8 lines
Hello, list,
I use 4 disk nodes with NBD.
All of my nodes have 2GB ram.But the buffer cache newer rise over 830MB.
Is there some limit?
For writes only 40% of the ram can be "dirty".
Quoted 1 line
Where can i change this limit? (if it is)
I believe there is a way to change it, I am pretty sure that it has been discussed on the kernel list a couple of months ago, I don't remember exactly what the change is, and I think the change was more complicated that was obvious.
The previous subject on a similar thing was:
"kernel 2.6.13 buffer strangeness"Roger
Quoted 35 lines
Thanks,
Janos[root@st-0001 root]# free
total used free shared
buffers cached
Mem: 2073152 933188 1139964 0
836776 43416
-/+ buffers/cache: 52996 2020156
Swap: 0 0 0
[root@st-0001 root]# cat/proc/meminfoMemTotal: 2073152 kB MemFree: 1139012 kB Buffers: 835928 kB Cached: 43448 kB SwapCached: 0 kB Active: 12872 kB Inactive: 871424 kB HighTotal: 1179584 kB HighFree: 1129764 kB LowTotal: 893568 kB LowFree: 9248 kB SwapTotal: 0 kB SwapFree: 0 kB Dirty: 0 kB Writeback: 0 kB Mapped: 9104 kB Slab: 30248 kB CommitLimit: 1036576 kB Committed_AS: 15428 kB PageTables: 408 kB VmallocTotal: 114680 kB VmallocUsed: 196 kB VmallocChunk: 114476 kBQuoted 9 lines
[root@st-0001 root]#
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected] More majordomo info at
http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
A message that is not in the archive (sent off-list or not yet received): [email protected]
----- Original Message -----
From: Nate Diller
To: JaniD++
Cc: Roger Heflin ; [email protected]
Sent: Friday, December 23, 2005 9:58 AM
Subject: Re: buffer cache question
On 11/21/05, JaniD++ <[email protected]> wrote:
Hi,
I have read back in the kernel-archives, and found this messages, about the same theme, but there is one difference!
On the old messages:
Quoted 8 lines
Nate Diller wrote:
just found the culprit. guess i should have read the code the first time.get_dirty_limits()indrivers/block/page_writeback.chas a hard-coded upper limit todirty_ratio. it's capped to half of the unmapped pages, so maybe 30-40% of your system's memory. so if you are brave, just remove the "/ 2" parts from the 'if (dirty_ratio>unmapped_ratio/ 2)dirty_ratio=unmapped_ratio/ 2;' check, and you can have all the OOM goodness you want.
...
Quoted 6 lines
I changed that bit of code to:
if (
dirty_ratio>unmapped_ratio- 10)
dirty_ratio=unmapped_ratio- 10;and added a couple of sanity checks so that it couldn't get below 5 or
above 95.
Quoted 2 lines
Then set
/proc/sys/vm/dirty_ratioto 95 anddirty_background_ratioto 1.
In this case, this modification is only for the *dirty* memory buffer. I want to use more buffer *cache*! :-)
The unwritten dirty memory ratio is good enough for me.
Anybody has an idea?
<snip>
Quoted 24 lines
Quoted 24 lines
[root@st-0001 root]# free
total used free shared
buffers cached
Mem: 2073152 933188 1139964 0
836776 43416
-/+ buffers/cache: 52996 2020156
Swap: 0 0 0
[root@st-0001 root]# cat/proc/meminfoMemTotal: 2073152 kB MemFree: 1139012 kB Buffers: 835928 kB Cached: 43448 kB SwapCached: 0 kB Active: 12872 kB Inactive: 871424 kB HighTotal: 1179584 kB HighFree: 1129764 kB LowTotal: 893568 kB LowFree: 9248 kB SwapTotal: 0 kB SwapFree: 0 kB Dirty: 0 kB Writeback: 0 kB Mapped: 9104 kB
Quoted 8 lines
Quoted 8 lines
Slab: 30248 kB CommitLimit: 1036576 kB Committed_AS: 15428 kB PageTables: 408 kB VmallocTotal: 114680 kB VmallocUsed: 196 kB VmallocChunk: 114476 kB[root@st-0001 root]#
looks like you're barely using any of your high memory. maybe NBD doesn't have highmem support. what file system are you using?
NATE
I cannot understant this.
NBD need to support highmem for buffering?
If know right, the kernel does buffering, not NBD!
But the kernel only use ~830MB for buffer cache instead of dinamically use all free memory like page cache.
This is one raw disk node, independent from file system.
Cheers,
Janos
Quoted 13 lines
----- Original Message -----
From: Nate Diller
To: JaniD++
Cc: Roger Heflin ; [email protected]
Sent: Friday, December 23, 2005 9:58 AM
Subject: Re: buffer cache questionlooks like you're barely using any of your high memory. maybe NBD doesn't have highmem support. what file system are you using?
NATE
Quoted 8 lines
I cannot understant this.
NBD need to support highmem for buffering?
If know right, the kernel does buffering, not NBD!
But the kernel only use ~830MB for buffer cache instead of dinamically use all free memory like page cache.This is one raw disk node, independent from file system.
this is an NBD client, using CONFIG_BLK_DEV_NBD? on the 2.6 series kernel? if so, then it, like any other kernel component using the page cache, needs to explicity use kmap/kunmap to make use of memory in the high memory zone. on a 32 bit machine, any pages above the 896 meg mark are treated specially inside the linux kernel (see
http://kerneltrap.org/node/2450).
if you don't have highmem support (CONFIG_HIGHMEM4G) enabled, then enabling that should fix it. if you already have it enabled (it looked like it to me, based on your /proc/meminfo) then there is a bug somewhere.
it would seem from a brief inspection that the send/recv_bvec functions in nbd (2.6.13) do use kmap. I don't know the nbd code very well, it seems that Pavel Machek wrote the code, he or block layer maintainer Jens Axboe may know something I don't. So if enabling highmem in your .config doesn't help, try CC'ing them with your issue.
in the mean time, one of the memory split patches, such as the 4G:4G
patch, should get things working.
NATE
Hi,
----- Original Message -----
From: "Nate Diller" <[email protected]>
To: "JaniD++" <[email protected]>
Cc: <[email protected]>; <[email protected]>
Sent: Thursday, December 29, 2005 5:10 AM
Subject: Re: buffer cache question
Quoted 8 lines
Quoted 8 lines
----- Original Message -----
From: Nate Diller
To: JaniD++
Cc: Roger Heflin ; [email protected]
Sent: Friday, December 23, 2005 9:58 AM
Subject: Re: buffer cache questionlooks like you're barely using any of your high memory. maybe NBD
doesn't
Quoted 9 lines
Quoted 9 lines
have highmem support. what file system are you using?
NATE
I cannot understant this.
NBD need to support highmem for buffering?
If know right, the kernel does buffering, not NBD!
But the kernel only use ~830MB for buffer cache instead of dinamically
use
Quoted 11 lines
Quoted 4 lines
all free memory like page cache.
This is one raw disk node, independent from file system.
this is an NBD client, using
CONFIG_BLK_DEV_NBD? on the 2.6 series kernel? if so, then it, like any other kernel component using the page cache, needs to explicity use kmap/kunmap to make use of memory in the high memory zone. on a 32 bit machine, any pages above the 896 meg mark are treated specially inside the linux kernel (see
http://kerneltrap.org/node/2450).
Yes, i have read this helpful document, but there is about the memory usage, mapping, and not exactly the buffering/caching.
Yes, i use the 2.6 series NBD.
Quoted 5 lines
if you don't have highmem support (
CONFIG_HIGHMEM4G) enabled, then enabling that should fix it. if you already have it enabled (it looked like it to me, based on your/proc/meminfo) then there is a bug somewhere.
Yes, it is enabled.
Quoted 8 lines
it would seem from a brief inspection that the send/recv_bvec functions in nbd (2.6.13) do use kmap. I don't know the nbd code very well, it seems that Pavel Machek wrote the code, he or block layer maintainer Jens Axboe may know something I don't. So if enabling highmem in your .config doesn't help, try CC'ing them with your issue.
in the mean time, one of the memory split patches, such as the 4G:4G
patch, should get things working.
The NBD use buffer cache (i am sure), but the highest limit is about 830-840M.
Where can i get the split patch?
Thanks for helpful answers!
Cheers,
Janos
Quoted 2 lines
NATE