Skip to contentSkip to conversation

buffer cache question

DISCUSSION JaniD++ 3 participants 6 messages Dec 19, 2005 – Dec 29, 2005 1 message not in the archive

lore
Share & tools
lore

JaniD++

author
#0 lore raw

Hello, list,

I use 4 disk nodes with NBD.
All of my nodes have 2GB ram.

But the buffer cache newer rise over 830MB.

Is there some limit?
Where can i change this limit? (if it is)

Thanks,
Janos

[root@st-0001 root]# free
total used free shared buffers cached
Mem: 2073152 933188 1139964 0 836776 43416
-/+ buffers/cache: 52996 2020156
Swap: 0 0 0
[root@st-0001 root]# cat /proc/meminfo

MemTotal:      2073152 kB
MemFree:       1139012 kB
Buffers:        835928 kB
Cached:          43448 kB
SwapCached:          0 kB
Active:          12872 kB
Inactive:       871424 kB
HighTotal:     1179584 kB
HighFree:      1129764 kB
LowTotal:       893568 kB
LowFree:          9248 kB
SwapTotal:           0 kB
SwapFree:            0 kB
Dirty:               0 kB
Writeback:           0 kB
Mapped:           9104 kB
Slab:            30248 kB
CommitLimit:   1036576 kB
Committed_AS:    15428 kB
PageTables:        408 kB
VmallocTotal:   114680 kB
VmallocUsed:       196 kB
VmallocChunk:   114476 kB

[root@st-0001 root]#

#1 lore raw
Quoted 9 lines

-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of JaniD++
Sent: Sunday, December 18, 2005 3:09 PM
To: [email protected]
Subject: buffer cache question

Quoted 8 lines

Hello, list,

I use 4 disk nodes with NBD.
All of my nodes have 2GB ram.

But the buffer cache newer rise over 830MB.

Is there some limit?

For writes only 40% of the ram can be "dirty".

Quoted 1 line

Where can i change this limit? (if it is)

I believe there is a way to change it, I am pretty sure that it has been discussed on the kernel list a couple of months ago, I don't remember exactly what the change is, and I think the change was more complicated that was obvious.

The previous subject on a similar thing was:
"kernel 2.6.13 buffer strangeness"

Roger

Quoted 44 lines

Thanks,
Janos

[root@st-0001 root]# free
total used free shared
buffers cached
Mem: 2073152 933188 1139964 0
836776 43416
-/+ buffers/cache: 52996 2020156
Swap: 0 0 0
[root@st-0001 root]# cat /proc/meminfo

MemTotal:      2073152 kB
MemFree:       1139012 kB
Buffers:        835928 kB
Cached:          43448 kB
SwapCached:          0 kB
Active:          12872 kB
Inactive:       871424 kB
HighTotal:     1179584 kB
HighFree:      1129764 kB
LowTotal:       893568 kB
LowFree:          9248 kB
SwapTotal:           0 kB
SwapFree:            0 kB
Dirty:               0 kB
Writeback:           0 kB
Mapped:           9104 kB
Slab:            30248 kB
CommitLimit:   1036576 kB
Committed_AS:    15428 kB
PageTables:        408 kB
VmallocTotal:   114680 kB
VmallocUsed:       196 kB
VmallocChunk:   114476 kB

[root@st-0001 root]#

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected] More majordomo info at
http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

author
#2 lore raw

Hi,

I have read back in the kernel-archives, and found this messages, about the same theme, but there is one difference!

On the old messages:

Quoted 8 lines

Nate Diller wrote:
just found the culprit. guess i should have read the code the first time. get_dirty_limits() in drivers/block/page_writeback.c has a hard-coded upper limit to dirty_ratio. it's capped to half of the unmapped pages, so maybe 30-40% of your system's memory. so if you are brave, just remove the "/ 2" parts from the 'if (dirty_ratio > unmapped_ratio / 2) dirty_ratio = unmapped_ratio / 2;' check, and you can have all the OOM goodness you want.

...

Quoted 6 lines

I changed that bit of code to:

if (dirty_ratio > unmapped_ratio - 10)
dirty_ratio = unmapped_ratio - 10;

and added a couple of sanity checks so that it couldn't get below 5 or

above 95.

Quoted 2 lines

Then set /proc/sys/vm/dirty_ratio to 95 and dirty_background_ratio to 1.

In this case, this modification is only for the *dirty* memory buffer. I want to use more buffer *cache*! :-)
The unwritten dirty memory ratio is good enough for me.

Anybody has an idea?

Cheers,
Janos

----- Original Message -----
From: "Roger Heflin" <[email protected]>
To: "'JaniD++'" <[email protected]>; <[email protected]>
Sent: Monday, December 19, 2005 7:15 PM
Subject: RE: buffer cache question

Quoted 82 lines
Quoted 9 lines

-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of JaniD++
Sent: Sunday, December 18, 2005 3:09 PM
To: [email protected]
Subject: buffer cache question

Quoted 8 lines

Hello, list,

I use 4 disk nodes with NBD.
All of my nodes have 2GB ram.

But the buffer cache newer rise over 830MB.

Is there some limit?

For writes only 40% of the ram can be "dirty".

Quoted 1 line

Where can i change this limit? (if it is)

I believe there is a way to change it, I am pretty sure that it has been discussed on the kernel list a couple of months ago, I don't remember exactly what the change is, and I think the change was more complicated that was obvious.

The previous subject on a similar thing was:
"kernel 2.6.13 buffer strangeness"

Roger

Quoted 35 lines

Thanks,
Janos

[root@st-0001 root]# free
total used free shared
buffers cached
Mem: 2073152 933188 1139964 0
836776 43416
-/+ buffers/cache: 52996 2020156
Swap: 0 0 0
[root@st-0001 root]# cat /proc/meminfo

MemTotal:      2073152 kB
MemFree:       1139012 kB
Buffers:        835928 kB
Cached:          43448 kB
SwapCached:          0 kB
Active:          12872 kB
Inactive:       871424 kB
HighTotal:     1179584 kB
HighFree:      1129764 kB
LowTotal:       893568 kB
LowFree:          9248 kB
SwapTotal:           0 kB
SwapFree:            0 kB
Dirty:               0 kB
Writeback:           0 kB
Mapped:           9104 kB
Slab:            30248 kB
CommitLimit:   1036576 kB
Committed_AS:    15428 kB
PageTables:        408 kB
VmallocTotal:   114680 kB
VmallocUsed:       196 kB
VmallocChunk:   114476 kB
Quoted 9 lines

[root@st-0001 root]#

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected] More majordomo info at
http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

author
#4 lore raw
In reply to a message not in the archive

----- Original Message -----
From: Nate Diller
To: JaniD++
Cc: Roger Heflin ; [email protected]
Sent: Friday, December 23, 2005 9:58 AM
Subject: Re: buffer cache question

On 11/21/05, JaniD++ <[email protected]> wrote:
Hi,

I have read back in the kernel-archives, and found this messages, about the same theme, but there is one difference!

On the old messages:

Quoted 8 lines

Nate Diller wrote:
just found the culprit. guess i should have read the code the first time. get_dirty_limits() in drivers/block/page_writeback.c has a hard-coded upper limit to dirty_ratio. it's capped to half of the unmapped pages, so maybe 30-40% of your system's memory. so if you are brave, just remove the "/ 2" parts from the 'if (dirty_ratio > unmapped_ratio / 2) dirty_ratio = unmapped_ratio / 2;' check, and you can have all the OOM goodness you want.

...

Quoted 6 lines

I changed that bit of code to:

if (dirty_ratio > unmapped_ratio - 10)
dirty_ratio = unmapped_ratio - 10;

and added a couple of sanity checks so that it couldn't get below 5 or

above 95.

Quoted 2 lines

Then set /proc/sys/vm/dirty_ratio to 95 and dirty_background_ratio to 1.

In this case, this modification is only for the *dirty* memory buffer. I want to use more buffer *cache*! :-)
The unwritten dirty memory ratio is good enough for me.

Anybody has an idea?

<snip>

Quoted 24 lines
Quoted 24 lines

[root@st-0001 root]# free
total used free shared
buffers cached
Mem: 2073152 933188 1139964 0
836776 43416
-/+ buffers/cache: 52996 2020156
Swap: 0 0 0
[root@st-0001 root]# cat /proc/meminfo

MemTotal:      2073152 kB
MemFree:       1139012 kB
Buffers:        835928 kB
Cached:          43448 kB
SwapCached:          0 kB
Active:          12872 kB
Inactive:       871424 kB
HighTotal:     1179584 kB
HighFree:      1129764 kB
LowTotal:       893568 kB
LowFree:          9248 kB
SwapTotal:           0 kB
SwapFree:            0 kB
Dirty:               0 kB
Writeback:           0 kB
Mapped:           9104 kB
Quoted 8 lines
Quoted 8 lines
Slab:            30248 kB
CommitLimit:   1036576 kB
Committed_AS:    15428 kB
PageTables:        408 kB
VmallocTotal:   114680 kB
VmallocUsed:       196 kB
VmallocChunk:   114476 kB

[root@st-0001 root]#

looks like you're barely using any of your high memory. maybe NBD doesn't have highmem support. what file system are you using?

NATE

I cannot understant this.
NBD need to support highmem for buffering?
If know right, the kernel does buffering, not NBD!
But the kernel only use ~830MB for buffer cache instead of dinamically use all free memory like page cache.

This is one raw disk node, independent from file system.

Cheers,
Janos

#5 lore raw
Quoted 13 lines

----- Original Message -----
From: Nate Diller
To: JaniD++
Cc: Roger Heflin ; [email protected]
Sent: Friday, December 23, 2005 9:58 AM
Subject: Re: buffer cache question

looks like you're barely using any of your high memory. maybe NBD doesn't have highmem support. what file system are you using?

NATE

Quoted 8 lines

I cannot understant this.
NBD need to support highmem for buffering?
If know right, the kernel does buffering, not NBD!
But the kernel only use ~830MB for buffer cache instead of dinamically use all free memory like page cache.

This is one raw disk node, independent from file system.

this is an NBD client, using CONFIG_BLK_DEV_NBD? on the 2.6 series kernel? if so, then it, like any other kernel component using the page cache, needs to explicity use kmap/kunmap to make use of memory in the high memory zone. on a 32 bit machine, any pages above the 896 meg mark are treated specially inside the linux kernel (see
http://kerneltrap.org/node/2450).

if you don't have highmem support (CONFIG_HIGHMEM4G) enabled, then enabling that should fix it. if you already have it enabled (it looked like it to me, based on your /proc/meminfo) then there is a bug somewhere.

it would seem from a brief inspection that the send/recv_bvec functions in nbd (2.6.13) do use kmap. I don't know the nbd code very well, it seems that Pavel Machek wrote the code, he or block layer maintainer Jens Axboe may know something I don't. So if enabling highmem in your .config doesn't help, try CC'ing them with your issue.
in the mean time, one of the memory split patches, such as the 4G:4G
patch, should get things working.

NATE

author
#6 lore raw

Hi,

----- Original Message -----
From: "Nate Diller" <[email protected]>
To: "JaniD++" <[email protected]>
Cc: <[email protected]>; <[email protected]>
Sent: Thursday, December 29, 2005 5:10 AM
Subject: Re: buffer cache question

Quoted 8 lines
Quoted 8 lines

----- Original Message -----
From: Nate Diller
To: JaniD++
Cc: Roger Heflin ; [email protected]
Sent: Friday, December 23, 2005 9:58 AM
Subject: Re: buffer cache question

looks like you're barely using any of your high memory. maybe NBD

doesn't

Quoted 9 lines
Quoted 9 lines

have highmem support. what file system are you using?

NATE

I cannot understant this.
NBD need to support highmem for buffering?
If know right, the kernel does buffering, not NBD!
But the kernel only use ~830MB for buffer cache instead of dinamically

use

Quoted 11 lines
Quoted 4 lines

all free memory like page cache.

This is one raw disk node, independent from file system.

this is an NBD client, using CONFIG_BLK_DEV_NBD? on the 2.6 series kernel? if so, then it, like any other kernel component using the page cache, needs to explicity use kmap/kunmap to make use of memory in the high memory zone. on a 32 bit machine, any pages above the 896 meg mark are treated specially inside the linux kernel (see
http://kerneltrap.org/node/2450).

Yes, i have read this helpful document, but there is about the memory usage, mapping, and not exactly the buffering/caching.
Yes, i use the 2.6 series NBD.

Quoted 5 lines

if you don't have highmem support (CONFIG_HIGHMEM4G) enabled, then enabling that should fix it. if you already have it enabled (it looked like it to me, based on your /proc/meminfo) then there is a bug somewhere.

Yes, it is enabled.

Quoted 8 lines

it would seem from a brief inspection that the send/recv_bvec functions in nbd (2.6.13) do use kmap. I don't know the nbd code very well, it seems that Pavel Machek wrote the code, he or block layer maintainer Jens Axboe may know something I don't. So if enabling highmem in your .config doesn't help, try CC'ing them with your issue.
in the mean time, one of the memory split patches, such as the 4G:4G
patch, should get things working.

The NBD use buffer cache (i am sure), but the highest limit is about 830-840M.

Where can i get the split patch?

Thanks for helpful answers!

Cheers,
Janos

Quoted 2 lines

NATE