Skip to contentSkip to conversation

Limit the size of the pagecache

RFC Al Boldi 3 participants 7 messages Jan 25, 2007

lore
Share & tools
lore

Al Boldi

author
#0 lore raw
Quoted 9 lines · Rik van Riel
Quoted 3 lines · Christoph Lameter

This is a patch using some of Aubrey's work plugging it in what is IMHO the right way. Feel free to improve on it. I have gotten repeatedly requests to be able to limit the pagecache.

IMHO it's a bad hack.

It would be better to identify the problem this "feature" is trying to fix, and then fix the root cause.

Ok, here is the problem: kswapd.

Limiting the page-cache memory inhibits invoking kswapd needlessly, aiding performance and easing OOM pressures.

I tried the patch; it works.

But it needs a bit of debugging. Setting pagecache_ratio = 1 either deadlocks or reduces thru-put to < 1mb/s.

Thanks!

--
Al

#1 lore raw
Quoted 7 lines · Al Boldi
Quoted 6 lines · Rik van Riel
Quoted 3 lines · Christoph Lameter

This is a patch using some of Aubrey's work plugging it in what is IMHO the right way. Feel free to improve on it. I have gotten repeatedly requests to be able to limit the pagecache.

IMHO it's a bad hack.

Quoted 7 lines
Quoted 2 lines

It would be better to identify the problem this "feature" is trying to fix, and then fix the root cause.

Ok, here is the problem: kswapd.

Limiting the page-cache memory inhibits invoking kswapd needlessly, aiding performance and easing OOM pressures.

Apart from kswapd, limiting pagecache helps performance of applications by not eating away their ANON pages or other parts of its resident data set. When there is enough free memory, then there is no performance issue. However memory is always utilized to the max. Hence every pagecache page that is allocated should come from some application's RSS, or from cold pagecache page. If that page was stolen from some application, then that application pays the price for swapping or reading the page back to memory. This scenario is what we
want to avoid. All that we are trying to achieve is that pagecache
eats a (unmapped) pagecache page and not steal memory from other important application's resident set.

Certainly this should be a configurable option and kernel's behavior should not be changed in general.

Quoted 1 line

I tried the patch; it works.

:)

Quoted 2 lines

But it needs a bit of debugging. Setting pagecache_ratio = 1 either deadlocks or reduces thru-put to < 1mb/s.

Yes, going below 5% on my 1GB RAM machine causes severe performance problems. We need to hard wire a reasonable lower limit and not provide a noose for the end user to tie around!

--Vaidy

Quoted 12 lines

Thanks!

--
Al

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

#2 lore raw
Quoted 13 lines

Apart from kswapd, limiting pagecache helps performance of applications by not eating away their ANON pages or other parts of its resident data set. When there is enough free memory, then there is no performance issue. However memory is always utilized to the max. Hence every pagecache page that is allocated should come from some application's RSS, or from cold pagecache page. If that page was stolen from some application, then that application pays the price for swapping or reading the page back to memory. This scenario is what we
want to avoid. All that we are trying to achieve is that pagecache
eats a (unmapped) pagecache page and not steal memory from other important application's resident set.

Quoted 2 lines

Certainly this should be a configurable option and kernel's behavior should not be changed in general.

Ah, this would be a clear case of the page reclaim selecting the wrong working set.

It is perfectly fine for a page cache page to evict a app page (be it anon or not) if that page cache page is used more frequently than the app page in question.

Trouble seems to be that the current algorithm gets it quite wrong at times.

Also stating that free memory somehow is good for you is weird, free memory is a loss, you under utilise your machine. Keeping clean pagecache pages in there that are likely to be referenced again is a clear win; it saves the tediously slow load from disk.

So you're now proposing to limit the page cache were as its clear that the better solution would be to tune replacement policy (and or provide hints to said mechanism using madvise/fadvise)

#3 lore raw
Quoted 15 lines · Peter Zijlstra
Quoted 12 lines

Apart from kswapd, limiting pagecache helps performance of applications by not eating away their ANON pages or other parts of its resident data set. When there is enough free memory, then there is no performance issue. However memory is always utilized to the max. Hence every pagecache page that is allocated should come from some application's RSS, or from cold pagecache page. If that page was stolen from some application, then that application pays the price for swapping or reading the page back to memory. This scenario is what we
want to avoid. All that we are trying to achieve is that pagecache
eats a (unmapped) pagecache page and not steal memory from other important application's resident set.

Quoted 2 lines

Certainly this should be a configurable option and kernel's behavior should not be changed in general.

Quoted 6 lines

Ah, this would be a clear case of the page reclaim selecting the wrong working set.

It is perfectly fine for a page cache page to evict a app page (be it anon or not) if that page cache page is used more frequently than the app page in question.

Well, this is true only as long as all applications running in the system are graded equally and it is kernel's job to provide the best of the system resources to all applications.

Quoted 2 lines

Trouble seems to be that the current algorithm gets it quite wrong at times.

The current reclaim code does a good job based on the assumption that pages belonging to different applications have equal priority. The aging of the page is independent of application's priority or class. This is good for best overall system performance.

The new use case that is challenging this assumption is the fact that application groups fall into different class on the same system and there is a need to make certain class perform better at the cost of certain other class of applications. In this scenario system performance is not judged by overall average throughput, but by performance of certain class of applications only.

A backup job running in the database server can take any amount of performance hit to marginally improve database performance since that is what the users care about. We would run into similar situations when running various virtualization and consolidation solutions.

Quoted 4 lines

Also stating that free memory somehow is good for you is weird, free memory is a loss, you under utilise your machine. Keeping clean pagecache pages in there that are likely to be referenced again is a clear win; it saves the tediously slow load from disk.

Agreed

Quoted 4 lines

So you're now proposing to limit the page cache were as its clear that the better solution would be to tune replacement policy (and or provide hints to said mechanism using madvise/fadvise)

Well, we may need to use both the approach. Hints with madvise/fadvise is definitely a good approach and the kernel should take these hints aggressively. Yet even with these hints we may want to have limits in the interest of other applications that do not use pagecache.

System wide limit to pagecache may not sound very interesting, but if we think about 'containers' and group of process having such limits it will have more practical use cases. An aggregation of process having
limit on pagecache would give relative importance to certain class of
pages during page replacement. Controlling limits among group of applications will help achieve peak application performance with the applications that we care about.

--Vaidy

author
#4 lore raw
In reply to Peter Zijlstra
Quoted 12 lines · Peter Zijlstra
Quoted 12 lines

Apart from kswapd, limiting pagecache helps performance of applications by not eating away their ANON pages or other parts of its resident data set. When there is enough free memory, then there is no performance issue. However memory is always utilized to the max. Hence every pagecache page that is allocated should come from some application's RSS, or from cold pagecache page. If that page was stolen from some application, then that application pays the price for swapping or reading the page back to memory. This scenario is what we
want to avoid. All that we are trying to achieve is that pagecache
eats a (unmapped) pagecache page and not steal memory from other important application's resident set.

Quoted 5 lines
Quoted 2 lines

Certainly this should be a configurable option and kernel's behavior should not be changed in general.

Ah, this would be a clear case of the page reclaim selecting the wrong working set.

Yes.

Quoted 3 lines

It is perfectly fine for a page cache page to evict a app page (be it anon or not) if that page cache page is used more frequently than the app page in question.

It seems, that there is currently a clear preference for pagecache-page over app-page. Some form of prio-selection could probably aid the situation.

Quoted 2 lines

Trouble seems to be that the current algorithm gets it quite wrong at times.

It breaks down when memory gets tight. You can actually hear it thrashing the disk, although it's not supposed to thrash, even with swapoff.

Quoted 4 lines

Also stating that free memory somehow is good for you is weird, free memory is a loss, you under utilise your machine. Keeping clean pagecache pages in there that are likely to be referenced again is a clear win; it saves the tediously slow load from disk.

That's the theory.

Quoted 1 line

So you're now proposing to limit the page cache

As a workaround.

Quoted 2 lines

where as its clear that
the better solution would be to tune replacement policy

Yes. Hopefully successfully.

Quoted 2 lines

(and or provide
hints to said mechanism using madvise/fadvise)

Not feasible; source is sometimes not immediately available.

Thanks!

--
Al

author
#5 lore raw
In reply to Vaidyanathan Srinivasan
Quoted 17 lines · Vaidyanathan Srinivasan
Quoted 8 lines · Al Boldi
Quoted 7 lines · Rik van Riel
Quoted 3 lines · Christoph Lameter

This is a patch using some of Aubrey's work plugging it in what is IMHO the right way. Feel free to improve on it. I have gotten repeatedly requests to be able to limit the pagecache.

IMHO it's a bad hack.

Quoted 7 lines
Quoted 2 lines

It would be better to identify the problem this "feature" is trying to fix, and then fix the root cause.

Ok, here is the problem: kswapd.

Limiting the page-cache memory inhibits invoking kswapd needlessly, aiding performance and easing OOM pressures.

Quoted 11 lines

Apart from kswapd, limiting pagecache helps performance of applications by not eating away their ANON pages or other parts of its resident data set. When there is enough free memory, then there is no performance issue. However memory is always utilized to the max. Hence every pagecache page that is allocated should come from some application's RSS, or from cold pagecache page. If that page was stolen from some application, then that application pays the price for swapping or reading the page back to memory. This scenario is what we
want to avoid. All that we are trying to achieve is that pagecache
eats a (unmapped) pagecache page and not steal memory from other important application's resident set.

Agreed 100%. Thanks for expanding exactly what I meant.

Quoted 5 lines

Certainly this should be a configurable option and kernel's behavior should not be changed in general.

Quoted 2 lines

I tried the patch; it works.

Quoted 8 lines

:)
:

Quoted 2 lines

But it needs a bit of debugging. Setting pagecache_ratio = 1 either deadlocks or reduces thru-put to < 1mb/s.

Yes, going below 5% on my 1GB RAM machine causes severe performance problems. We need to hard wire a reasonable lower limit and not provide a noose for the end user to tie around!

One reason to test full range settings, is to expose underlying system problems, like scalability. By limiting the range, you only hide a problem that was exposed.

Thanks!

--
Al

#6 lore raw
Quoted 38 lines · Al Boldi
Quoted 14 lines · Vaidyanathan Srinivasan
Quoted 5 lines · Al Boldi
Quoted 4 lines · Rik van Riel
Quoted 3 lines · Christoph Lameter

This is a patch using some of Aubrey's work plugging it in what is IMHO the right way. Feel free to improve on it. I have gotten repeatedly requests to be able to limit the pagecache.

Quoted 8 lines
Quoted 4 lines

IMHO it's a bad hack.

It would be better to identify the problem this "feature" is trying to fix, and then fix the root cause.

Ok, here is the problem: kswapd.

Limiting the page-cache memory inhibits invoking kswapd needlessly, aiding performance and easing OOM pressures.

Quoted 11 lines

Apart from kswapd, limiting pagecache helps performance of applications by not eating away their ANON pages or other parts of its resident data set. When there is enough free memory, then there is no performance issue. However memory is always utilized to the max. Hence every pagecache page that is allocated should come from some application's RSS, or from cold pagecache page. If that page was stolen from some application, then that application pays the price for swapping or reading the page back to memory. This scenario is what we
want to avoid. All that we are trying to achieve is that pagecache
eats a (unmapped) pagecache page and not steal memory from other important application's resident set.

Agreed 100%. Thanks for expanding exactly what I meant.

Quoted 9 lines

Certainly this should be a configurable option and kernel's behavior should not be changed in general.

Quoted 2 lines

I tried the patch; it works.

:)
:

Quoted 2 lines

But it needs a bit of debugging. Setting pagecache_ratio = 1 either deadlocks or reduces thru-put to < 1mb/s.

Quoted 7 lines
Quoted 3 lines

Yes, going below 5% on my 1GB RAM machine causes severe performance problems. We need to hard wire a reasonable lower limit and not provide a noose for the end user to tie around!

One reason to test full range settings, is to expose underlying system problems, like scalability. By limiting the range, you only hide a problem that was exposed.

Agreed. This is a good point.

Quoted 12 lines

Thanks!

--
Al

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/