dcache and rmap
Subsystems (3)
RFC PATCH 1/1 dcache and rmap
Hi,
I got tired of finding my box with 50-60% percent of memory tied up in dentry/inode caches every morning after update-db runs or after doing a find / -name "*" to generate a list of files for backups. So I decided to make a stab at fixing this.
The problem is that when there is not much memory pressure the vm is happy to let the above caches expand and expand... What I did was factored the shrink calls out of do_try_to_free_pages and placed an additional call to shrink in kswapd which can get called if kswapd does not need to use do_try_to_free_pages.
The issue then becomes when to call the new shrink_caches function? I changed the dcache logic to estimate and track the number of new pages alloced to dentries. Once a
threshold is exceeded, kswapd calls shrink_caches. Using a threshold of 32 pages works
well here.
Patch is against 19-pre7-ac2.
Comments?
Ed Tomlinson
# This is a BitKeeper generated patch for the following project:
# Project Name: Linux kernel tree
# This patch format is intended for GNU patch command version 2.5 or higher. # This patch includes the following deltas:
# ChangeSet 1.422 -> 1.425 # fs/dcache.c 1.18 -> 1.21 # mm/vmscan.c 1.60 -> 1.61
# include/linux/dcache.h 1.11 -> 1.12
#
# The following is the BitKeeper ChangeSet Log
# --------------------------------------------
# 02/05/04 [email protected] 1.423
# Modify the cache shrinking logic to apply pressure when the dcache
# grows by more than <n> pages (currently 32). Note the slab growth # detection code, though not perfect, is okay for this use. # -------------------------------------------- # 02/05/04 [email protected] 1.424 # Fix comments # -------------------------------------------- # 02/05/04 [email protected] 1.425 # grammar # -------------------------------------------- # diff -Nru a/fs/dcache.c b/fs/dcache.c --- a/fs/dcache.c Sat May 4 23:17:36 2002 +++ b/fs/dcache.c Sat May 4 23:17:36 2002 @@ -305,6 +305,22 @@ spin_lock(&dcache_lock); }
+
+/**
+ * Have we allocated over n pages worth of
+ * dentries entries?
+ */
+
+#define DENTRIES_PER_PAGE (PAGE_SIZE/(sizeof(struct dentry)+8)) +
+static int dcache_alloc_count = 0;
+
+int try_to_shrink(int pages)
+{
+ return dcache_alloc_count > pages;
+}
+
+
/**
* prune_dcache - shrink the dcache
* @count: number of entries to try and free
@@ -567,7 +583,13 @@
count = dentry_stat.nr_unused / priority;
- prune_dcache(count);
+ prune_dcache(count);
+
+ /*
+ * relieve some pressure...
+ */
+ dcache_alloc_count /= 2;
+
return kmem_cache_shrink_nr(dentry_cache);
}
@@ -585,8 +607,17 @@
struct dentry * d_alloc(struct dentry * parent, const struct qstr *name) {
+ static int nr_entry_base = 0;
char * str;
struct dentry *dentry;
+
+ if (dentry_stat.nr_dentry < nr_entry_base)
+ nr_entry_base = dentry_stat.nr_dentry;
+
+ if (dentry_stat.nr_dentry-nr_entry_base > DENTRIES_PER_PAGE) {
+ dcache_alloc_count++;
+ nr_entry_base = dentry_stat.nr_dentry;
+ }
dentry = kmem_cache_alloc(dentry_cache, GFP_KERNEL); if (!dentry)
include/linux/dcache.h +1 -0
@@ -173,6 +173,7 @@/* dcache memory management */ extern int shrink_dcache_memory (int ,unsigned int );extern void prune_dcache (int ); +extern int try_to_shink (int );/* icache memory management (defined in linux/fs/inode.c) */ extern int shrink_icache_memory (int ,int );
mm/vmscan.c +26 -5
@@ -562,6 +562,24 @@if (inactive_high (zone) >0 )refill_inactive_zone (zone, priority); } + +static int shrink_caches (unsigned int pri,unsigned int gfp_mask) +{ +int ret =0 ; + +/* +* Eat memory from +* dentry, inode and filesystem quota caches. +*/ + ret +=shrink_dcache_memory (pri, gfp_mask); + ret +=shrink_icache_memory (1 , gfp_mask); +#ifdef CONFIG_QUOTA + ret +=shrink_dqcache_memory (pri, gfp_mask); +#endif + +return ret; +} +/* * Worker function for kswapd and try_to_free_pages, we get
@@ -571,6 +589,7 @@* This function will also move pages to the inactive list, * if needed. */ +static int do_try_to_free_pages (unsigned int gfp_mask) {int ret =0 ;
@@ -580,11 +599,7 @@* dentry, inode and filesystem quota caches. */ ret +=page_launder (gfp_mask); - ret +=shrink_dcache_memory (DEF_PRIORITY , gfp_mask); - ret +=shrink_icache_memory (1 , gfp_mask); -#ifdef CONFIG_QUOTA - ret +=shrink_dqcache_memory (DEF_PRIORITY , gfp_mask); -#endif + ret +=shrink_caches (DEF_PRIORITY , gfp_mask);/* * Move pages from the active list to the inactive list.
@@ -653,6 +668,9 @@* If there are applications that are active memory-allocators * (most normal use), this basically shouldn't matter. */ + +#define DCACHE_PAGES (32) +int kswapd (void *unused) {struct task_struct *tsk = current;
@@ -691,6 +709,9 @@*/ if (free_high (ALL_ZONES ) >=0 ||free_low (ANY_ZONE ) >0 )do_try_to_free_pages (GFP_KSWAPD ); +else +if (try_to_shrink (DCACHE_PAGES )) +shrink_caches (DEF_PRIORITY ,GFP_KSWAPD );refill_freelist ();
-----------------------------
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/
Quoted 4 lines
I got tired of finding my box with 50-60% percent of memory tied up in dentry/inode caches every morning after update-db runs or after doing a find / -name "*" to generate a list of files for backups. So I decided to make a stab at fixing this.
Are you actually out of memory at this point, and they're consuming space you really need?
Quoted 5 lines
The issue then becomes when to call the new
shrink_cachesfunction? I changed the dcache logic to estimate and track the number of new pages alloced to dentries. Once a threshold is exceeded, kswapd
callsshrink_caches. Using a threshold of 32 pages works well
here.
Why would you want to shrink the cache if you had plenty of free memory? Maybe I'm mistunderstanding what you're trying to do, but that's what your explanation sounds like ....
M.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/
Quoted 7 lines · Martin J. Bligh
Quoted 4 lines
I got tired of finding my box with 50-60% percent of memory tied up in dentry/inode caches every morning after update-db runs or after doing a find / -name "*" to generate a list of files for backups. So I decided to make a stab at fixing this.
Are you actually out of memory at this point, and they're consuming space you really need?
Think of this another way. There are 100000+ dentry/inodes in memory comsuming 250M or so. Meanwhile load is light and the background aging is able to supply pages for the freelist. We do not reclaim this storage until we have vm pressure. Usually this pressure is artifical, if we had reclaimed the storage it would not have occured, our caches would have more useful data in them, and half the memory would not sit idle for half a day.
We age the rest of the memory to keep it hot. Rmap does a good job
and keeps the freelist heathly. In this case nothing ages the dentries and they get very cold. My code ensures that the memory consumed by the, potentially cold, dentries/inodes is not excessive.
(I hate getting paged at 3 in the morning)
Ed Tomlinson
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/
Quoted 16 lines
Quoted 7 lines
Quoted 4 lines
I got tired of finding my box with 50-60% percent of memory tied up in dentry/inode caches every morning after update-db runs or after doing a find / -name "*" to generate a list of files for backups. So I decided to make a stab at fixing this.
Are you actually out of memory at this point, and they're consuming space you really need?
Think of this another way. There are 100000+ dentry/inodes in memory comsuming 250M or so. Meanwhile load is light and the background aging is able to supply pages for the freelist. We do not reclaim this storage until we have vm pressure. Usually this pressure is artifical, if we had reclaimed the storage it would not have occured, our caches would have more useful data in them, and half the memory would not sit idle for half a day.
Quoted 4 lines
We age the rest of the memory to keep it hot. Rmap does a good job
and keeps the freelist heathly. In this case nothing ages the dentries and they get very cold. My code ensures that the memory consumed by the, potentially cold, dentries/inodes is not excessive.
If there's no pressure on memory, then using it for caches is a good thing. Why throw away data before we're out of space? If we are under pressure on memory then dcache should shrink easily and rapidly. If it's not, then make it shrink properly, don't just limit it to an arbitrary size that may be totally unsuitable for some workloads. You could even age it instead ... that'd make more sense than restricting it to a static size.
M.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/
Quoted 16 lines · Martin J. Bligh
Quoted 16 lines
Quoted 7 lines
Quoted 4 lines
I got tired of finding my box with 50-60% percent of memory tied up in dentry/inode caches every morning after update-db runs or after doing a find / -name "*" to generate a list of files for backups. So I decided to make a stab at fixing this.
Are you actually out of memory at this point, and they're consuming space you really need?
Think of this another way. There are 100000+ dentry/inodes in memory comsuming 250M or so. Meanwhile load is light and the background aging is able to supply pages for the freelist. We do not reclaim this storage until we have vm pressure. Usually this pressure is artifical, if we had reclaimed the storage it would not have occured, our caches would have more useful data in them, and half the memory would not sit idle for half a day.
Quoted 7 lines
Quoted 4 lines
We age the rest of the memory to keep it hot. Rmap does a good job
and keeps the freelist heathly. In this case nothing ages the dentries and they get very cold. My code ensures that the memory consumed by the, potentially cold, dentries/inodes is not excessive.If there's no pressure on memory, then using it for caches is a good thing. Why throw away data before we're out of space? If we are under
The point is there is always memory pressure. Sometimes kswapd can supply the pages needed without calling do_try_to_free_pages. When this happens the dcache/icache can grow since we never try to shrink it. My patch changes this.
Quoted 3 lines
pressure on memory then dcache should shrink easily and rapidly. If it's not, then make it shrink properly, don't just limit it to an arbitrary size that may be totally unsuitable for some workloads.
There is _no_ arbitrary limit. All that happens is that if the dcache grows
by more than n pages we try to shrink it once. If the system its actually
using the dentries, the dcache/icache will grow as needed. If its not using them, which is the case here more often than not, they get dropped and storage is freed to be better used by the rest of the vm.
Quoted 2 lines
You could even age it instead ... that'd make more sense than restricting it to a static size.
Again I apply pressure. There are no static limits. Aging would be nice but that would probably require rather massive changes to the way the slab cache works.
Ed Tomlinson
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/
Quoted 4 lines · Ed Tomlinson
I got tired of finding my box with 50-60% percent of memory tied up in dentry/inode caches every morning after update-db runs or after doing a find / -name "*" to generate a list of files for backups. So I decided to make a stab at fixing this.
I would only consider this a problem if the size of the cache were so poorly controlled that it would trigger pageout of in-core user data. Also, the cacheing algorithm doesn't seem to take into account turnover rates and expansion rate.. e.g. if turnover is too rapid, expand, if expansion is too fast, try directly reclaiming largely unused entries.
Quoted 5 lines · Ed Tomlinson
The problem is that when there is not much memory pressure the vm is happy to let the above caches expand and expand... What I did was factored the shrink calls out of >
do_try_to_free_pagesand placed an additional call to shrink in kswapd which can get called if kswapd does not need to usedo_try_to_free_pages.
Well, the VM doesn't really have anything to do with this; it's just a random structure floating out in filesystem code that needs notification of when memory gets low to shrink itself. But anyway, all the VM really cares about is memory is running low and it wants to get some from somewhere. Bad space behavior is either the cacheing algorithm's fault or its implementation's.
Abstracting this out to a "cache shrinking driver" might open up useful possibilities, for instance, dynamic registration for per-object caches.
Quoted 4 lines · Ed Tomlinson
The issue then becomes when to call the new
shrink_cachesfunction? I changed the dcache logic to estimate and track the number of new pages alloced to dentries. Once a threshold is exceeded, kswapd callsshrink_caches. Using a threshold of 32 pages works well here.
Well, I think there are three major design issues. The first is the magic number of 32 pages. This (compile-time!) tunable is almost guaranteed not to work for everyone. The second is that in order to address the issue you're actually concerned about, it seems you would have to present some method for caches to know their allocation requests would require evicting other useful data to satisfy; for instance, evicting pagecache holding useful (but clean) user data to allocate dcache. Third, the distinguished position of the dcache is suspicious to me; I feel that a greater degree of generality is in order.
In short, I don't think you went far enough. How do you feel about GFP_SPECULATIVE (a.k.a. GFP_DONT_TRY_TOO_HARD), cache priorities and cache shrinking drivers?
Cheers,
Bill
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/
On May 6, 2002 09:44 pm, William Lee Irwin III wrote
stuff omitted
Quoted 3 lines
Well, I think there are three major design issues. The first is the magic number of 32 pages. This (compile-time!) tunable is almost guaranteed not to work for everyone. The second is that in order to
Yes. I figured I would keep it simple for now. This could be made a
proc tuneable or kswapd could be modifed to autotune this.
Quoted 3 lines
address the issue you're actually concerned about, it seems you would have to present some method for caches to know their allocation requests would require evicting other useful data to satisfy; for instance,
This is an interesting idea. Might be as simple as hooking into the slab.c code and checking if the number of freepages is to small. Then we could increment a counter/flag (which counter/flag set at cache create time, associated with a 'driver'. See below). If this works it would be generic too...
Quoted 3 lines
evicting pagecache holding useful (but clean) user data to allocate dcache. Third, the distinguished position of the dcache is suspicious to me; I feel that a greater degree of generality is in order.
In two years I have never seen any cache other than the dcache/icache cause the problem. I short, yes its suspicious, but warented (here) by experience.
Quoted 3 lines
In short, I don't think you went far enough. How do you feel about
GFP_SPECULATIVE(a.k.a.GFP_DONT_TRY_TOO_HARD), cache priorities and cache shrinking drivers?
Think I will sprinkle slab.c with a printk or two to see if we detect when it's allocations are eating other caches. If this works we should be able to let the vm know when to shrink the slab cache and to let it know which caches need shrinking (ie shrink_caches becomes a 'driver' to shrink the dcache/icache family. kmem_cache_reap being the generic 'driver')
Thanks for the feedback and interesting idea,
Ed Tomlinson
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/
Quoted 3 lines · At some point in the past, I wrote:
Quoted 3 lines
In short, I don't think you went far enough. How do you feel about
GFP_SPECULATIVE(a.k.a.GFP_DONT_TRY_TOO_HARD), cache priorities and cache shrinking drivers?
Quoted 6 lines · Ed Tomlinson
Think I will sprinkle
slab.cwith a printk or two to see if we detect when it's allocations are eating other caches. If this works we should be able to let the vm know when to shrink the slab cache and to let it know which caches need shrinking (ieshrink_cachesbecomes a 'driver' to shrink the dcache/icache family.kmem_cache_reapbeing the generic 'driver') Thanks for the feedback and interesting idea,
Well, the trick is kmem_cache_reap() doesn't know how to prune references to things within the cache like prune_dcache() does. It is in essence its own cache in front of another cache for allocations. I'm not sure making kmem_cache_reap() trigger reaping of the caches it's parked in front of is a great idea. It seems that it would go the other
direction: reaping a cache parked in front of a slab would want to call kmem_cache_reap() sometime afterward (so the memory is actually reclaimed instead of sitting in the slab cache). IIRC the VM actually does this at some point after calling the assorted cache shrink functions. kmem_cache_reap() may well be needed in contexts where the caches are doing fine jobs of keeping their space under control or shrinking themselves just fine, without intervention from outside callers.
Cheers,
Bill
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/
Quoted 10 lines · William Lee Irwin III
in essence its own cache in front of another cache for allocations. I'm not sure making
kmem_cache_reap()trigger reaping of the caches it's parked in front of is a great idea. It seems that it would go the other
direction: reaping a cache parked in front of a slab would want to callkmem_cache_reap()sometime afterward (so the memory is actually reclaimed instead of sitting in the slab cache). IIRC the VM actually does this at some point after calling the assorted cache shrink functions.kmem_cache_reap()may well be needed in contexts where the caches are doing fine jobs of keeping their space under control or shrinking themselves just fine, without intervention from outside callers.
<hint>
In newer Solaris versions (at least SunOS 5.7/5.8) kmem_cache_t has a new method to allow reclaiming of objects on memory pressure. </hint>
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/
Quoted 4 lines · Christoph Hellwig
<hint>
In newer Solaris versions (at least SunOS 5.7/5.8)kmem_cache_thas a new method to allow reclaiming of objects on memory pressure. </hint>
Well, it can't be that bad of an idea then. I still like keeping the things separate, but maybe there just isn't that much difference.
Cheers,
Bill
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/
Quoted 13 lines · William Lee Irwin III
Quoted 3 lines · At some point in the past, I wrote:
Quoted 3 lines
In short, I don't think you went far enough. How do you feel about
GFP_SPECULATIVE(a.k.a.GFP_DONT_TRY_TOO_HARD), cache priorities and cache shrinking drivers?Quoted 6 lines · Ed Tomlinson
Think I will sprinkle
slab.cwith a printk or two to see if we detect when it's allocations are eating other caches. If this works we should be able to let the vm know when to shrink the slab cache and to let it know which caches need shrinking (ieshrink_cachesbecomes a 'driver' to shrink the dcache/icache family.kmem_cache_reapbeing the generic 'driver') Thanks for the feedback and interesting idea,
Quoted 12 lines
Well, the trick is
kmem_cache_reap()doesn't know how to prune references to things within the cache likeprune_dcache()does. It is in essence its own cache in front of another cache for allocations. I'm not sure makingkmem_cache_reap()trigger reaping of the caches it's parked in front of is a great idea. It seems that it would go the other
direction: reaping a cache parked in front of a slab would want to callkmem_cache_reap()sometime afterward (so the memory is actually reclaimed instead of sitting in the slab cache). IIRC the VM actually does this at some point after calling the assorted cache shrink functions.kmem_cache_reap()may well be needed in contexts where the caches are doing fine jobs of keeping their space under control or shrinking themselves just fine, without intervention from outside callers.
I remember Linus once mentioned an idea that slab pages should have ->writepage() that triggers shrink of the front-end cache.
Along this way, VM would just manage one big cache---physical memory.
Quoted 4 lines
Cheers,
Bill
Nikita.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/
Hi,
I did something similiar in the patch I posted under the subject:
[RFC][PATCH] cache shrinking via page age
Only I used the same method we now use to shrink caches but triggered them using page aging, and at the same time making the trigger cache specific.
Another though I had was to put the 'freeable' slab pages onto the inactive clean list and reclaim them when they reach the head of the list. It gets a
little tricky since slabs can contain multiple pages... Before trying this
I want to see how well what I have posted works.
Ed Tomlinson
Quoted 31 lines · Nikita Danilov
Quoted 13 lines · William Lee Irwin III
Quoted 3 lines · At some point in the past, I wrote:
Quoted 3 lines
In short, I don't think you went far enough. How do you feel about
GFP_SPECULATIVE(a.k.a.GFP_DONT_TRY_TOO_HARD), cache priorities and cache shrinking drivers?Quoted 6 lines · Ed Tomlinson
Think I will sprinkle
slab.cwith a printk or two to see if we detect when it's allocations are eating other caches. If this works we should be able to let the vm know when to shrink the slab cache and to let it know which caches need shrinking (ieshrink_cachesbecomes a 'driver' to shrink the dcache/icache family.kmem_cache_reapbeing the generic 'driver') Thanks for the feedback and interesting idea,Quoted 13 lines
Well, the trick is
kmem_cache_reap()doesn't know how to prune references to things within the cache likeprune_dcache()does. It is in essence its own cache in front of another cache for allocations. I'm not sure makingkmem_cache_reap()trigger reaping of the caches it's parked in front of is a great idea. It seems that it would go the other
direction: reaping a cache parked in front of a slab would want to callkmem_cache_reap()sometime afterward (so the memory is actually reclaimed instead of sitting in the slab cache). IIRC the VM actually does this at some point after calling the assorted cache shrink functions.kmem_cache_reap()may well be needed in contexts where the caches are doing fine jobs of keeping their space under control or shrinking themselves just fine, without intervention from outside callers.I remember Linus once mentioned an idea that slab pages should have ->writepage() that triggers shrink of the front-end cache.
Quoted 6 lines
Along this way, VM would just manage one big cache---physical memory.
Quoted 2 lines
Cheers,
BillNikita.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/
Quoted 7 lines · Ed Tomlinson
Hi,
I did something similiar in the patch I posted under the subject:
[RFC][PATCH] cache shrinking via page age
Quoted 8 lines
Only I used the same method we now use to shrink caches but triggered them using page aging, and at the same time making the trigger cache specific.
Another though I had was to put the 'freeable' slab pages onto the inactive clean list and reclaim them when they reach the head of the list. It gets a
little tricky since slabs can contain multiple pages... Before trying this
I want to see how well what I have posted works.
Using the VM would be better...
It means that you'll need to create an address_space (and possibly, at this stage, an inode) to back the slab pages.
Probably there is no need to create a radix tree, nor to give those pages an ->index. Just bump page->count to indicate that the page is "sort-of" in the pagecache. The kernel presently assumes, in __remove_inode_page() and __add_to_page_cache() that these pages are in a radix-tree. I don't think it needs to. Just set slab_space.page_tree to NULL and handle that in the various places which go oops ;)
Probably there is no need to create a mapping per slab. A global one should suffice.
There's no need to ever set those pages dirty, hence there's no need for a ->writepage.
When pages are added to the slab you should put some slab backpointer into page->private, set PG_private and increment page->count.
Most of the work will be in slab_space->a_ops.releasepage(). In there you'll need to find the slab via page->private and just start tossing away pages until you see the target page come "free". Then clear PG_private and drop page->count and return success from ->releasepage(). The page is still "in the pagecache" so it has an incremented ->count. The modified __remove_inode_page() will perform the final release.
That's all fairly straightforward. The tricky bit is getting the aging right. These pages don't have referenced bits in the pte's. Possibly, running mark_page_accessed() inside kmem_cache_alloc would be sufficient. It would be more accurate to make every user of a slab object "touch" that object's backing page but that's not feasible.
-
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/