using page aging to shrink caches

Hi,

I have never been happy with the way slab cache shrinking worked. This is an attempt to make it better. Working with the rmap vm on pre7-ac2, I have done the following.

1. Moved reapable slab pages on to the active and inactive dirty lists.
2. When slab pages enter the inactive dirty list I count the number of pages seen on a per cache basis.
3. If slab pages manage to reach the front of the inactive dirty list I count the pages seen on a per cache basis.
4. After inactive_refill/inactive_refill_zone calls I scan the slab caches and, via a callback, shrink the caches using the number of pages from 3 & 4 as a goal for each cache.
5. kmem_cache_reap is called as a last ditch effort before declaring a oom.
6. Since slab pages are kernel map via 8M pages on i386, the hardware page reference bit is fairly much useless. When a slab is created it pages are marked as referenced. I also mark pages in the lookup/get functions for inodes, dentries, dquota and buffer_heads.

A few comments.

#1 avoids the need to create mappings for the slab pages by intercepting them in page_launder_zone and refill_inactive_zone before we start really playing with the pages.

#2 was done to avoid having 500,000 plus dentry/inodes on a lightly loaded system. Seems with low vm pressure rmap can supply enough free pages via background aging. This process avoids page_launder but does call refill_inactive...

#4 uses ends up using kmem_cache_shrink_nr to shrink caches. For the caches that require pruning this call gets wraped. The calling sequence for icache, dcache and dquoto shrinking now only prunes when it cannot get enought pages via a simple shrink.

Comments?
Ed Tomlinson

PS. I may not see replies until Tuesday.

-----------
# This is a BitKeeper generated patch for the following project:
# Project Name: Linux kernel tree
# This patch format is intended for GNU patch command version 2.5 or higher. # This patch includes the following deltas:

#	           ChangeSet	1.422   -> 1.432  
#	         fs/buffer.c	1.63    -> 1.64   
#	         fs/dcache.c	1.18    -> 1.24   
#	          fs/dquot.c	1.18    -> 1.22   
#	         mm/vmscan.c	1.60    -> 1.68   
#	include/linux/slab.h	1.9     -> 1.14   

# include/linux/dcache.h 1.11 -> 1.14
# mm/slab.c 1.16 -> 1.22
# fs/inode.c 1.35 -> 1.41
#
# The following is the BitKeeper ChangeSet Log
# --------------------------------------------
# 02/05/10 [email protected] 1.423
# Use the vm's page aging to tell us when we need to shrink the caches. # The vm uses callbacks to tell the slabs caches its time to shrink. # --------------------------------------------
# 02/05/10 [email protected] 1.424
# Change the way process_shrinks is called so refill_invalid does not

# need to be changed.
# --------------------------------------------
# 02/05/11	[email protected]	1.426
# Simplify the scheme.  Use per cache callbacks instead of per family.
# This lets us target specific caches instead of being generic.  We
# still include a generic call (kmem_cache_reap) as a failsafe
# before ooming.
# --------------------------------------------
# 02/05/11	[email protected]	1.428
# Change factoring, removing changes from background_aging and putting
# the kmem_call_shrinkers call in kswapd.
# --------------------------------------------
# 02/05/11	[email protected]	1.429
# Base the number of pages a cache is shrunk on the number of pages the
# vm sees in refill_inactive_zone instead of on a magic priority.
# --------------------------------------------
# 02/05/12	[email protected]	1.430
# improve shrink methods for dcache, dquota and icache
# --------------------------------------------
# 02/05/12	[email protected]	1.428.1.1
# The icache is a slave of the dcache.  We will not reuse the inodes so
# lets clean them all.
# --------------------------------------------
# 02/05/12	[email protected]	1.428.1.2
# Only call shrink callback if we have seen a slab's worth of pages
# --------------------------------------------
# 02/05/13	[email protected]	1.428.1.3
# Andrew Morton pointed out that kernal pages are big (8M) and the 
# hardware reference bit is working with these big pages.  This makes 
# aging slabs on 4K pages a little more difficult.  Andrew suggested 
# hooking into the kmem_cache_alloc process and set the bit(s) there.  
# This changeset does this.
# --------------------------------------------
# 02/05/16	[email protected]	1.428.1.6
# Improve aging for dcache, inode and dquota pages by setting the ref
# bits in the various lookup/get code.
# --------------------------------------------
# 02/05/16	[email protected]	1.428.1.7
# Add another kmem_touch_page in getblk for buffer_heads.  Convert
# kmem_touch_page to a macro.
# --------------------------------------------
# 02/05/17	[email protected]	1.432
# Use the number of slab pages that enter or are requeued in the 
# inactive dirty list as the goal for the number of pages to shrink 
# a slab cache.  Each cache can have its own shrink callback, though
# only caches that need 'pruning' require specialized functions.
# --------------------------------------------
#
diff -Nru a/fs/buffer.c b/fs/buffer.c
--- a/fs/buffer.c	Fri May 17 23:36:37 2002
+++ b/fs/buffer.c	Fri May 17 23:36:37 2002
@@ -1059,8 +1059,10 @@
 		struct buffer_head * bh;

bh = get_hash_table(dev, block, size);
- if (bh)
+ if (bh) {
+ kmem_touch_page(bh);
return bh;
+ }

if (!grow_buffers(dev, block, size))
free_more_memory();

diff -Nru a/fs/dcache.c b/fs/dcache.c
@@ -538,18 +538,11 @@
 
 /*
  * This is called from kswapd when we think we need some
- * more memory, but aren't really sure how much. So we
- * carefully try to free a _bit_ of our dcache, but not
- * too much.
- *
- * Priority:
- *   0 - very urgent: shrink everything
- *  ...
- *   6 - base-level: try to shrink a bit.
+ * more memory. 
  */
-int shrink_dcache_memory(int priority, unsigned int gfp_mask)
+int shrink_dcache_memory(kmem_cache_t *cachep, int pages, int priority, int gfp_mask)
 {
-	int count = 0;
+	int count = kmem_cache_shrink_nr(cachep, pages);
 
 	/*
 	 * Nasty deadlock avoidance.

@@ -563,12 +556,13 @@
 	 * block allocations, but for now:
 	 */
 	if (!(gfp_mask & __GFP_FS))
-		return 0;
+		return count;
 
-	count = dentry_stat.nr_unused / priority;
-
-	prune_dcache(count);
-	return kmem_cache_shrink_nr(dentry_cache);
+	if (count < pages) {
+		prune_dcache(dentry_stat.nr_unused/priority);
+		count += kmem_cache_shrink_nr(cachep, pages-count);
+	}
+	return count;
 }
 
 #define NAME_ALLOC_LEN(len)	((len+16) & ~15)

@@ -730,6 +724,7 @@
 		}
 		__dget_locked(dentry);
 		dentry->d_vfs_flags |= DCACHE_REFERENCED;
+		kmem_touch_page(dentry);
 		spin_unlock(&dcache_lock);
 		return dentry;
 	}

@@ -1186,6 +1181,8 @@
 	if (!dentry_cache)
 		panic("Cannot create dentry cache");
 
+	kmem_set_shrinker(dentry_cache, (shrinker_t)shrink_dcache_memory);
+
 #if PAGE_SHIFT < 13
 	mempages >>= (13 - PAGE_SHIFT);
 #endif

@@ -1278,6 +1275,9 @@
 			SLAB_HWCACHE_ALIGN, NULL, NULL);
 	if (!dquot_cachep)
 		panic("Cannot create dquot SLAB cache");
+	
+	kmem_set_shrinker(dquot_cachep, (shrinker_t)shrink_dqcache_memory);
+	
 #endif
 
 	dcache_init(mempages);

diff -Nru a/fs/dquot.c b/fs/dquot.c
@@ -1024,12 +1024,17 @@
 	}
 }
 
-int shrink_dqcache_memory(int priority, unsigned int gfp_mask)
+int shrink_dqcache_memory(kmem_cache_t *cachep, int pages, int priority, unsigned int gfp_mask)
 {
-	lock_kernel();
-	prune_dqcache(nr_free_dquots / (priority + 1));
-	unlock_kernel();
-	return kmem_cache_shrink_nr(dquot_cachep);
+	int count = kmem_cache_shrink_nr(cachep, pages);
+
+	if (count < pages) {
+		lock_kernel();
+		prune_dqcache(nr_free_dquots);
+		unlock_kernel();
+		count += kmem_cache_shrink_nr(cachep, pages-count);
+	}
+	return count;
 }
 
 /*

@@ -1148,6 +1153,7 @@
 #endif
 	dquot->dq_referenced++;
 	dqstats.lookups++;
+	kmem_touch_page(dquot);
 
 	return dquot;
 }

diff -Nru a/fs/inode.c b/fs/inode.c
@@ -193,6 +193,7 @@
 
 static inline void __iget(struct inode * inode)
 {
+	kmem_touch_page(inode);
 	if (atomic_read(&inode->i_count)) {
 		atomic_inc(&inode->i_count);
 		return;

@@ -708,9 +709,9 @@
 		schedule_task(&unused_inodes_flush_task);
 }
 
-int shrink_icache_memory(int priority, int gfp_mask)
+int shrink_icache_memory(kmem_cache_t *cachep, int pages, int priority, int gfp_mask)
 {
-	int count = 0;
+	int count = kmem_cache_shrink_nr(cachep, pages);
 
 	/*
 	 * Nasty deadlock avoidance..

@@ -720,12 +721,13 @@
 	 * in clear_inode() and friends..
 	 */
 	if (!(gfp_mask & __GFP_FS))
-		return 0;
+		return count;
 
-	count = inodes_stat.nr_unused / priority;
-
-	prune_icache(count);
-	return kmem_cache_shrink_nr(inode_cachep);
+	if (count < pages) {
+		prune_icache(inodes_stat.nr_unused);
+		count += kmem_cache_shrink_nr(cachep, pages-count);
+	}
+	return count;
 }
 
 /*

@@ -1172,6 +1174,8 @@
 					 NULL);
 	if (!inode_cachep)
 		panic("cannot create inode slab cache");
+
+	kmem_set_shrinker(inode_cachep, (shrinker_t)shrink_icache_memory);
 
 	unused_inodes_flush_task.routine = try_to_sync_unused_inodes;
 }

diff -Nru a/include/linux/dcache.h b/include/linux/dcache.h
@@ -171,15 +171,10 @@
 #define shrink_dcache() prune_dcache(0)
 struct zone_struct;
 /* dcache memory management */
-extern int shrink_dcache_memory(int, unsigned int);
 extern void prune_dcache(int);
 
 /* icache memory management (defined in linux/fs/inode.c) */
-extern int shrink_icache_memory(int, int);
 extern void prune_icache(int);
-
-/* quota cache memory management (defined in linux/fs/dquot.c) */
-extern int shrink_dqcache_memory(int, unsigned int);
 
 /* only used at mount-time */
 extern struct dentry * d_alloc_root(struct inode *);

diff -Nru a/include/linux/slab.h b/include/linux/slab.h
@@ -55,7 +55,27 @@
 				       void (*)(void *, kmem_cache_t *, unsigned long));
 extern int kmem_cache_destroy(kmem_cache_t *);
 extern int kmem_cache_shrink(kmem_cache_t *);
-extern int kmem_cache_shrink_nr(kmem_cache_t *);
+
+typedef int (*shrinker_t)(kmem_cache_t *, int, int, int);
+
+extern void kmem_set_shrinker(kmem_cache_t *, shrinker_t);
+extern int kmem_call_shrinkers(int, int);
+extern void kmem_count_page(struct page *);
+#define kmem_touch_page(addr) 		SetPageReferenced(virt_to_page(addr));
+
+/* shrink drivers */
+extern int kmem_shrink_pages(kmem_cache_t *, int, int, int);
+
+/* dcache shrinker ( defined in linux/fs/dcache.c) */
+extern int shrink_dcache_memory(kmem_cache_t *, int, int, int);
+
+/* icache shrinker (defined in linux/fs/inode.c) */
+extern int shrink_icache_memory(kmem_cache_t *, int, int, int);
+
+/* quota cache shrinker (defined in linux/fs/dquot.c) */
+extern int shrink_dqcache_memory(kmem_cache_t *, int, int, int);
+
+extern int kmem_cache_shrink_nr(kmem_cache_t *, int);
 extern void *kmem_cache_alloc(kmem_cache_t *, int);
 extern void kmem_cache_free(kmem_cache_t *, void *);
 

-----------
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/

Thank you! This is should help greatly with some of the vm imbalances by making slab reclaim part of the self tuning dynamics instead of hard coded magic numbers. Do you have any plans to port this patch to 2.5 for inclusion? It would be useful to get testing in the 2.5 before merging in 2.4.

-ben

"You will be reincarnated as a toad; and you will be much happier."
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/

Here is an improved version of the patch for pre8-ac5.

This moves things towards having the vm do the work of freeing the pages. I do wonder if it worth the effort in that slab pages are a bit different from other pages and get treated a little differently. For instance, we sometimes free slab pages in refill_inactive. Without this the caches can grow and grow without any possibility of shrinking when under low loads. By allowing freeing we avoid getting into a situation where slab pages cause an artificial shortage.

Finding a good method of handling the dcache/icache and dquota caches has been fun... What I do now is factor the pruning and shrinking into different calls. The puning, in effect, ages entries in the above caches. The rate I prune is simply the rate I see entries for these slabs in refill_inactive_zone. This is seems fair and, in my testing, works better than anything else I have tried (I have have experimented quite a bit). It also avoid using any magic numbers and is self tuning.

The logic has also be improved to free specific slabs instead of just freeing <n> freeable slabs when <n> were encountered by the vm. Now we try to free the slabs we encounter as we find them (see kmem_shrink_slab).

It works well on my UP box running pre8-ac5 without problems.

Think this is ready for wider testing. If any of you have test boxes that are having
vm problems, especially if they are slab related, it would be interesting to see if this helps.

Comments, questions and feedback very welcome,

Ed Tomlinson

Summary of changes.

fs/buffer.c
touch the object's page when a buffer head is looked up

fs/dcache.c
prune_cache now ages <n> entries instead of freeing <n> shrink_dcache_memory becomes age_dcache_memory and is called from the vm
using kmem_do_prunes.
set the pruner call backs for the dcache and dqcache
touch the object's page when a dentry is looked up

fs/dquot.c
shrink_dqcache_memory becomes age_dqcache_memory and is called from the vm. touch the objects's page when a dquot is found

fs/inode.c
touch the object's page when a inode is found
prune_inodes now ages <n> inodes instead of freeing <n>. shrink_icache_memory becomes age_icache_memory and is called from the vm. set the pruner call back for the icache to age_icache_memory.

include/linux/slab.h
add types, macros, and externs needed for the slab pages in lru scheme. To
rationalise includes, the externs for the age_<x>_memory calls move here from dcache.h

mm/slab.c
add pruner and count to kmem_cache_t and set them up when creating a cache add kmem_set_pruner to set the pruner call back
add kmem_count_page to count the entries in a slab page in cachep->count add keme_do_prunes to call the pruner call backs to age the slab cache entries add and remove reapable slab pages to the lru
add kmem_shrink_slab to remove a slab and free its memory if possible touch the page when allocating an entry in a slab

mm/vmscan.c
bug if we hit a slab page in the wrong list
handle slab pages in page_launder_zone. We either free the slab, if we cannot and
the slab's cache is growing we requeue it in the inactive list otherwise we move the page to the active list.
handle slab pages in refill_inactive_zone. We count the entries on the page for
kmem_do_prunes which gets called later. If the slab page is eligible to be
moved to the inactive list, first try to free it, if this fails move it to the inactive list. refill_inactive_zone now returns the number of pages freed.
in do_try_to_free_pages account for the pages freed by refill_inactive and call
kmem_do_prunes to age the slab caches in lock step with the vm scanning. A last ditch kmem_cache_reap is left before we conclude we are oom.
in kswapd call kmem_do_prunes after each background_aging call.

--------------
# This is a BitKeeper generated patch for the following project:
# Project Name: Linux kernel tree
# This patch format is intended for GNU patch command version 2.5 or higher. # This patch includes the following deltas:

#	           ChangeSet	1.406   -> 1.408  
#	         fs/buffer.c	1.66    -> 1.67   
#	         fs/dcache.c	1.19    -> 1.20   
#	          fs/dquot.c	1.18    -> 1.19   
#	         mm/vmscan.c	1.60    -> 1.62   
#	           mm/slab.c	1.16    -> 1.18   
#	include/linux/slab.h	1.8     -> 1.9    

# include/linux/dcache.h 1.11 -> 1.12
# fs/inode.c 1.36 -> 1.37
#
# The following is the BitKeeper ChangeSet Log
# --------------------------------------------
# 02/05/23 [email protected] 1.407

# age_pressure_v7.diff
# --------------------------------------------
# 02/05/23	[email protected]	1.408
# Fix the locking in vmscan for slab pages in lru.  Improve the comments
# too.
# --------------------------------------------
#
diff -Nru a/fs/buffer.c b/fs/buffer.c
--- a/fs/buffer.c	Thu May 23 21:38:19 2002
+++ b/fs/buffer.c	Thu May 23 21:38:19 2002
@@ -951,8 +951,10 @@
 		struct buffer_head * bh;

bh = get_hash_table(dev, block, size);
- if (bh)
+ if (bh) {
+ kmem_touch_page(bh);
return bh;
+ }

if (!grow_buffers(dev, block, size))
free_more_memory();

diff -Nru a/fs/dcache.c b/fs/dcache.c
@@ -321,7 +321,7 @@
 void prune_dcache(int count)
 {
 	spin_lock(&dcache_lock);
-	for (;;) {
+	for (; count ; count--) {
 		struct dentry *dentry;
 		struct list_head *tmp;
 

@@ -345,8 +345,6 @@
 			BUG();
 
 		prune_one_dentry(dentry);
-		if (!--count)
-			break;
 	}
 	spin_unlock(&dcache_lock);
 }

@@ -538,19 +536,10 @@
 
 /*
  * This is called from kswapd when we think we need some
- * more memory, but aren't really sure how much. So we
- * carefully try to free a _bit_ of our dcache, but not
- * too much.
- *
- * Priority:
- *   0 - very urgent: shrink everything
- *  ...
- *   6 - base-level: try to shrink a bit.
+ * more memory. 
  */
-int shrink_dcache_memory(int priority, unsigned int gfp_mask)
+int age_dcache_memory(kmem_cache_t *cachep, int entries, int gfp_mask)
 {
-	int count = 0;
-
 	/*
 	 * Nasty deadlock avoidance.
 	 *

@@ -565,10 +554,11 @@
 	if (!(gfp_mask & __GFP_FS))
 		return 0;
 
-	count = dentry_stat.nr_unused / priority;
+	if (entries > dentry_stat.nr_unused)
+		entries = dentry_stat.nr_unused;
 
-	prune_dcache(count);
-	return kmem_cache_shrink(dentry_cache);
+	prune_dcache(entries);
+	return entries;
 }
 
 #define NAME_ALLOC_LEN(len)	((len+16) & ~15)

@@ -730,6 +720,7 @@
 		}
 		__dget_locked(dentry);
 		dentry->d_vfs_flags |= DCACHE_REFERENCED;
+		kmem_touch_page(dentry);
 		spin_unlock(&dcache_lock);
 		return dentry;
 	}

@@ -1186,6 +1177,8 @@
 	if (!dentry_cache)
 		panic("Cannot create dentry cache");
 
+	kmem_set_pruner(dentry_cache, (pruner_t)age_dcache_memory);
+
 #if PAGE_SHIFT < 13
 	mempages >>= (13 - PAGE_SHIFT);
 #endif

@@ -1279,6 +1272,9 @@
 			SLAB_HWCACHE_ALIGN, NULL, NULL);
 	if (!dquot_cachep)
 		panic("Cannot create dquot SLAB cache");
+	
+	kmem_set_pruner(dquot_cachep, (pruner_t)age_dqcache_memory);
+	
 #endif
 
 	dcache_init(mempages);

diff -Nru a/fs/dquot.c b/fs/dquot.c
@@ -1026,10 +1026,13 @@
 
 int shrink_dqcache_memory(int priority, unsigned int gfp_mask)
 {
+	if (entries > nr_free_dquots)
+		entries = nr_free_dquots;
+
 	lock_kernel();
-	prune_dqcache(nr_free_dquots / (priority + 1));
+	prune_dqcache(entries);
 	unlock_kernel();
-	return kmem_cache_shrink(dquot_cachep);
+	return entries;
 }
 
 /*

@@ -1148,6 +1151,7 @@
 #endif
 	dquot->dq_referenced++;
 	dqstats.lookups++;
+	kmem_touch_page(dquot);
 
 	return dquot;
 }

diff -Nru a/fs/inode.c b/fs/inode.c
@@ -193,6 +193,7 @@
 
 static inline void __iget(struct inode * inode)
 {
+	kmem_touch_page(inode);
 	if (atomic_read(&inode->i_count)) {
 		atomic_inc(&inode->i_count);
 		return;

@@ -672,10 +673,11 @@
 
 	count = 0;
 	entry = inode_unused.prev;
-	while (entry != &inode_unused)
-	{
+	for(; goal; goal--) {
 		struct list_head *tmp = entry;
 
+		if (entry == &inode_unused)
+			break;
 		entry = entry->prev;
 		inode = INODE(tmp);
 		if (inode->i_state & (I_FREEING|I_CLEAR|I_LOCK))

@@ -690,8 +692,6 @@
 		list_add(tmp, freeable);
 		inode->i_state |= I_FREEING;
 		count++;
-		if (!--goal)
-			break;
 	}
 	inodes_stat.nr_unused -= count;
 	spin_unlock(&inode_lock);

@@ -708,10 +708,8 @@
 		schedule_task(&unused_inodes_flush_task);
 }
 
-int shrink_icache_memory(int priority, int gfp_mask)
+int age_icache_memory(kmem_cache_t *cachep, int entries, int gfp_mask)
 {
-	int count = 0;
-
 	/*
 	 * Nasty deadlock avoidance..
 	 *

@@ -722,10 +720,11 @@
 	if (!(gfp_mask & __GFP_FS))
 		return 0;
 
-	count = inodes_stat.nr_unused / priority;
+	if (entries > inodes_stat.nr_unused)
+		entries = inodes_stat.nr_unused;
 
-	prune_icache(count);
-	return kmem_cache_shrink(inode_cachep);
+	prune_icache(entries);
+	return entries;
 }
 
 /*

@@ -1172,6 +1171,8 @@
 					 NULL);
 	if (!inode_cachep)
 		panic("cannot create inode slab cache");
+
+	kmem_set_pruner(inode_cachep, (pruner_t)age_icache_memory);
 
 	unused_inodes_flush_task.routine = try_to_sync_unused_inodes;
 }

diff -Nru a/include/linux/dcache.h b/include/linux/dcache.h
@@ -171,15 +171,10 @@
 #define shrink_dcache() prune_dcache(0)
 struct zone_struct;
 /* dcache memory management */
-extern int shrink_dcache_memory(int, unsigned int);
 extern void prune_dcache(int);
 
 /* icache memory management (defined in linux/fs/inode.c) */
-extern int shrink_icache_memory(int, int);
 extern void prune_icache(int);
-
-/* quota cache memory management (defined in linux/fs/dquot.c) */
-extern int shrink_dqcache_memory(int, unsigned int);
 
 /* only used at mount-time */
 extern struct dentry * d_alloc_root(struct inode *);

diff -Nru a/include/linux/slab.h b/include/linux/slab.h
@@ -55,6 +55,26 @@
 				       void (*)(void *, kmem_cache_t *, unsigned long));
 extern int kmem_cache_destroy(kmem_cache_t *);
 extern int kmem_cache_shrink(kmem_cache_t *);
+
+typedef int (*pruner_t)(kmem_cache_t *, int, int);
+
+extern void kmem_set_pruner(kmem_cache_t *, pruner_t);
+extern int kmem_do_prunes(int);
+extern void kmem_count_page(struct page *);
+#define kmem_touch_page(addr)                 SetPageReferenced(virt_to_page(addr));
+
+/* shrink a slab */
+extern int kmem_shrink_slab(struct page *);
+
+/* dcache prune ( defined in linux/fs/dcache.c) */
+extern int age_dcache_memory(kmem_cache_t *, int, int);
+
+/* icache prune (defined in linux/fs/inode.c) */
+extern int age_icache_memory(kmem_cache_t *, int, int);
+
+/* quota cache prune (defined in linux/fs/dquot.c) */
+extern int age_dqcache_memory(kmem_cache_t *, int, int);
+
 extern void *kmem_cache_alloc(kmem_cache_t *, int);
 extern void kmem_cache_free(kmem_cache_t *, void *);
 

--------------
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/

Just from a short look:

What about doing mark_page_accessed in kmem_touch_page? And please do a s/pruner_t/kmem_pruner_t/

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/

This kind of cache reclamation logic is so sorely needed it's unimaginable. I'm quite grateful for your efforts in this direction, and hope to be able to provide some assistance in testing soon.

Cheers,
Bill
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/

mark_page_accessed expects a page struct. kmem_touch_page takes an address in the page, converts it to a kernel address and then marks the page.

Yes. Done.

One other style question. I am not completely happy with kmem_shrink_slab. Think that instead of setting the reference bit I should probably do something like return:

-1 - cache is growing
0 - slab has inuse objects
n - pages were freed

Comments?
Ed Tomlinson

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/

Of course after the virt_to_page..

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/

Hi,

Here is an improved version of the patch. It fixes a race in kmem_freepages (I do not see why the race should not happen in straight ac) and makes the following changes:

Aging works a little differently for pruneable caches. For these caches we use pruning to do aging. The rate we prune is simpily the rate we see objects on pages processed by vmscan. For the all other caches vm aging is used. Without this change two aging methods we being applied to the dcache/icache. This favored their pages and the vm was quite slow to trim them at times.

Second, since there is almost no overhead (ie no disk access), when refill_inactive_zone sees a slab page it wants to free it releases the slab and moves the pages to the inactive clean list.

An interesting note. If I directly free the pages in kmem_freepages I run into a race.
It seems that freepages can be lost... To see the race I do the following.

find / -name "*" > /dev/null &
irman &
dbench 40 &

When irman start its memory stress test free pages are lost. Its very easy to see this using zlatko calusic's xmm utility. With my patch the number of kernel pages should be quite stable. When the race occurs they jump and proc/meminfo shows missing pages. This is on UP with no preempth.

Patch applies to pre8-ac5.

Comments?

Ed Tomlinson

-----------------
# This is a BitKeeper generated patch for the following project:
# Project Name: Linux kernel tree
# This patch format is intended for GNU patch command version 2.5 or higher. # This patch includes the following deltas:

#	           ChangeSet	1.406   -> 1.412  
#	         fs/buffer.c	1.66    -> 1.68   
#	         fs/dcache.c	1.19    -> 1.21   
#	          fs/dquot.c	1.18    -> 1.20   
#	         mm/vmscan.c	1.60    -> 1.65   
#	           mm/slab.c	1.16    -> 1.21   
#	include/linux/slab.h	1.8     -> 1.10   

# include/linux/dcache.h 1.11 -> 1.12
# fs/inode.c 1.36 -> 1.38
#
# The following is the BitKeeper ChangeSet Log
# --------------------------------------------
# 02/05/24 [email protected] 1.407

# age_pressure_v8.diff
# --------------------------------------------
# 02/05/24	[email protected]	1.408
# Remove side effect from kmem_shrink_slab and fix vmscan to use the new
# return codes.
# --------------------------------------------
# 02/05/24	[email protected]	1.409
# Simpilify - try lates algorythm without touches in lookups.
# --------------------------------------------
# 02/05/24	[email protected]	1.410
# Use either vm aging or prune call back aging, not both.  Keep slab
# pages on the active list.
# --------------------------------------------
# 02/05/28	[email protected]	1.411
# fix locking in slab to use pagemap_lru_lock when shrinking or growing
# a cache.  Use the inactive clean list when freeing a slab's pages.
# This avoid a race so the vm does not lose track of pages.
# --------------------------------------------
# 02/05/29	[email protected]	1.412
# Prevent bug in page_launder from being hit due to a dangling 
# referencebit.  Improve accounting in refill_inactive.
# --------------------------------------------
#
diff -Nru a/fs/dcache.c b/fs/dcache.c
--- a/fs/dcache.c	Wed May 29 07:35:03 2002
+++ b/fs/dcache.c	Wed May 29 07:35:03 2002
@@ -321,7 +321,7 @@
 void prune_dcache(int count)
 {
 	spin_lock(&dcache_lock);
-	for (;;) {
+	for (; count ; count--) {
 		struct dentry *dentry;
 		struct list_head *tmp;

@@ -345,8 +345,6 @@
BUG();

prune_one_dentry(dentry);
- if (!--count)
- break;
}
spin_unlock(&dcache_lock);
}
@@ -538,19 +536,10 @@

/*
* This is called from kswapd when we think we need some
- * more memory, but aren't really sure how much. So we
- * carefully try to free a _bit_ of our dcache, but not
- * too much.
- *
- * Priority:
- * 0 - very urgent: shrink everything
- * ...
- * 6 - base-level: try to shrink a bit.
+ * more memory.
*/
-int shrink_dcache_memory(int priority, unsigned int gfp_mask) +int age_dcache_memory(kmem_cache_t *cachep, int entries, int gfp_mask)
{
- int count = 0;
-
/*
* Nasty deadlock avoidance.
*
@@ -565,10 +554,11 @@
if (!(gfp_mask & __GFP_FS))
return 0;

- count = dentry_stat.nr_unused / priority;
+ if (entries > dentry_stat.nr_unused)
+ entries = dentry_stat.nr_unused;

- prune_dcache(count);
- return kmem_cache_shrink(dentry_cache);
+ prune_dcache(entries);
+ return entries;
}

#define NAME_ALLOC_LEN(len) ((len+16) & ~15)
@@ -1186,6 +1176,8 @@
if (!dentry_cache)
panic("Cannot create dentry cache");

+ kmem_set_pruner(dentry_cache, (kmem_pruner_t)age_dcache_memory);
+
#if PAGE_SHIFT < 13
mempages >>= (13 - PAGE_SHIFT);
#endif
@@ -1279,6 +1271,9 @@
SLAB_HWCACHE_ALIGN, NULL, NULL);
if (!dquot_cachep)
panic("Cannot create dquot SLAB cache");
+
+ kmem_set_pruner(dquot_cachep, (kmem_pruner_t)age_dqcache_memory);
+
#endif

dcache_init(mempages);

diff -Nru a/fs/dquot.c b/fs/dquot.c
@@ -1026,10 +1026,13 @@
 
 int shrink_dqcache_memory(int priority, unsigned int gfp_mask)
 {
+	if (entries > nr_free_dquots)
+		entries = nr_free_dquots;
+
 	lock_kernel();
-	prune_dqcache(nr_free_dquots / (priority + 1));
+	prune_dqcache(entries);
 	unlock_kernel();
-	return kmem_cache_shrink(dquot_cachep);
+	return entries;
 }
 
 /*

diff -Nru a/fs/inode.c b/fs/inode.c
@@ -672,10 +672,11 @@
 
 	count = 0;
 	entry = inode_unused.prev;
-	while (entry != &inode_unused)
-	{
+	for(; goal; goal--) {
 		struct list_head *tmp = entry;
 
+		if (entry == &inode_unused)
+			break;
 		entry = entry->prev;
 		inode = INODE(tmp);
 		if (inode->i_state & (I_FREEING|I_CLEAR|I_LOCK))

@@ -690,8 +691,6 @@
 		list_add(tmp, freeable);
 		inode->i_state |= I_FREEING;
 		count++;
-		if (!--goal)
-			break;
 	}
 	inodes_stat.nr_unused -= count;
 	spin_unlock(&inode_lock);

@@ -708,10 +707,8 @@
 		schedule_task(&unused_inodes_flush_task);
 }
 
-int shrink_icache_memory(int priority, int gfp_mask)
+int age_icache_memory(kmem_cache_t *cachep, int entries, int gfp_mask)
 {
-	int count = 0;
-
 	/*
 	 * Nasty deadlock avoidance..
 	 *

@@ -722,10 +719,11 @@
 	if (!(gfp_mask & __GFP_FS))
 		return 0;
 
-	count = inodes_stat.nr_unused / priority;
+	if (entries > inodes_stat.nr_unused)
+		entries = inodes_stat.nr_unused;
 
-	prune_icache(count);
-	return kmem_cache_shrink(inode_cachep);
+	prune_icache(entries);
+	return entries;
 }
 
 /*

@@ -1172,6 +1170,8 @@
 					 NULL);
 	if (!inode_cachep)
 		panic("cannot create inode slab cache");
+
+	kmem_set_pruner(inode_cachep, (kmem_pruner_t)age_icache_memory);
 
 	unused_inodes_flush_task.routine = try_to_sync_unused_inodes;
 }

diff -Nru a/include/linux/dcache.h b/include/linux/dcache.h
@@ -171,15 +171,10 @@
 #define shrink_dcache() prune_dcache(0)
 struct zone_struct;
 /* dcache memory management */
-extern int shrink_dcache_memory(int, unsigned int);
 extern void prune_dcache(int);
 
 /* icache memory management (defined in linux/fs/inode.c) */
-extern int shrink_icache_memory(int, int);
 extern void prune_icache(int);
-
-/* quota cache memory management (defined in linux/fs/dquot.c) */
-extern int shrink_dqcache_memory(int, unsigned int);
 
 /* only used at mount-time */
 extern struct dentry * d_alloc_root(struct inode *);

diff -Nru a/include/linux/slab.h b/include/linux/slab.h
@@ -55,6 +55,26 @@
 				       void (*)(void *, kmem_cache_t *, unsigned long));
 extern int kmem_cache_destroy(kmem_cache_t *);
 extern int kmem_cache_shrink(kmem_cache_t *);
+
+typedef int (*kmem_pruner_t)(kmem_cache_t *, int, int);
+
+extern void kmem_set_pruner(kmem_cache_t *, kmem_pruner_t);
+extern int kmem_do_prunes(int);
+extern int kmem_count_page(struct page *);
+#define kmem_touch_page(addr)                 SetPageReferenced(virt_to_page(addr));
+
+/* shrink a slab */
+extern int kmem_shrink_slab(struct page *);
+
+/* dcache prune ( defined in linux/fs/dcache.c) */
+extern int age_dcache_memory(kmem_cache_t *, int, int);
+
+/* icache prune (defined in linux/fs/inode.c) */
+extern int age_icache_memory(kmem_cache_t *, int, int);
+
+/* quota cache prune (defined in linux/fs/dquot.c) */
+extern int age_dqcache_memory(kmem_cache_t *, int, int);
+
 extern void *kmem_cache_alloc(kmem_cache_t *, int);
 extern void kmem_cache_free(kmem_cache_t *, void *);
 

-----------------

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/