cache shrinking via page age

When running under low vm pressure rmap does not shrink caches. This
happens since we only call do_try_to_free_pages when we have a shortage. On my box the combination of background_aging calling refill_inactive_zone is able to supply the pages needed. The end result of this the box acts sluggish, with about half my memory used by slab pages (dcache/icache). This does correct itself under pressure but it should never get into this state in the first place.

Idealy we want all pages to be about the same age. Having half the pages in the system 'cold' in the slab cache is not good - it implies the other pages are 'hotter' than they need to be.

To fix the situation I move reapable slab pages into the active list. When aging moves a page into the inactive dirty list I watch for slab pages and record the caches with old pages. After refill_inactive/background_aging ends I call a new function, kmem_call_shrinkers. This scans the list of slab caches and, via a callback, shrinks caches with old pages. Note that we never swap out slab pages they just cycle through active and inactive dirty lists.

The end result is that slab caches are shrunk selectivily when they have old 'cold' pages. I avoid adding any magic numbers to the vm and create a generic interface to allow creators of slab caches to supply the vm with a unique method to shrink their caches.

When testing this there is one side effect to remember. Using cat /proc/slabinfo references pages - this will tend to keep the slab pages warmer than they should be. Like in quantum theory, watching (to often) can change results.

I have testing on UP only - think the locking is ok though...

Patch is against 2.4.19-pre7-ac2

Comments?
Ed Tomlinson

------------
# This is a BitKeeper generated patch for the following project:
# Project Name: Linux kernel tree
# This patch format is intended for GNU patch command version 2.5 or higher. # This patch includes the following deltas:

#	           ChangeSet	1.422   -> 1.428
#	         fs/dcache.c	1.18    -> 1.20
#	         mm/vmscan.c	1.60    -> 1.65
#	include/linux/slab.h	1.9     -> 1.11
#	           mm/slab.c	1.16    -> 1.19
#	          fs/inode.c	1.35    -> 1.37

#
# The following is the BitKeeper ChangeSet Log
# --------------------------------------------
# 02/05/10 [email protected] 1.423
# Use the vm's page aging to tell us when we need to shrink the caches. # The vm uses callbacks to tell the slabs caches its time to shrink. # --------------------------------------------
# 02/05/10 [email protected] 1.424
# Change the way process_shrinks is called so refill_invalid does not

# need to be changed.
# --------------------------------------------
# 02/05/10	[email protected]	1.425
# Remove debuging stuff
# --------------------------------------------
# 02/05/11	[email protected]	1.426
# Simplify the scheme.  Use per cache callbacks instead of per family.
# This lets us target specific caches instead of being generic.  We
# still include a generic call (kmem_cache_reap) as a failsafe
# before ooming.
# --------------------------------------------
# 02/05/11	[email protected]	1.427
# Remove debugging printk
# --------------------------------------------
# 02/05/11	[email protected]	1.428
# Change factoring, removing changes from background_aging and putting
# the kmem_call_shrinkers call in kswapd.
# --------------------------------------------
#
diff -Nru a/fs/dcache.c b/fs/dcache.c

fs/dcache.c +5 -0

@@ -1186,6 +1186,8 @@
 	if (!dentry_cache)
 		panic("Cannot create dentry cache");
 
+	kmem_set_shrinker(dentry_cache, (shrinker_t)kmem_shrink_dcache);
+
 #if PAGE_SHIFT < 13
 	mempages >>= (13 - PAGE_SHIFT);
 #endif

@@ -1278,6 +1280,9 @@
 			SLAB_HWCACHE_ALIGN, NULL, NULL);
 	if (!dquot_cachep)
 		panic("Cannot create dquot SLAB cache");
+
+	kmem_set_shrinker(dquot_cachep, (shrinker_t)kmem_shrink_dquota);
+
 #endif
 
 	dcache_init(mempages);

diff -Nru a/fs/inode.c b/fs/inode.c

fs/inode.c +2 -0

@@ -1173,6 +1173,8 @@
 	if (!inode_cachep)
 		panic("cannot create inode slab cache");
 
+	kmem_set_shrinker(inode_cachep, (shrinker_t)kmem_shrink_icache);
+
 	unused_inodes_flush_task.routine = try_to_sync_unused_inodes;
 }
 

diff -Nru a/include/linux/slab.h b/include/linux/slab.h

include/linux/slab.h +13 -0

@@ -55,6 +55,19 @@
 				       void (*)(void *, kmem_cache_t *, unsigned long));
 extern int kmem_cache_destroy(kmem_cache_t *);
 extern int kmem_cache_shrink(kmem_cache_t *);
+
+typedef int (*shrinker_t)(kmem_cache_t *, int, int);
+
+extern void kmem_set_shrinker(kmem_cache_t *, shrinker_t);
+extern int kmem_call_shrinkers(int, int);
+extern void kmem_count_page(struct page *);
+
+/* shrink drivers */
+extern int kmem_shrink_default(kmem_cache_t *, int, int);
+extern int kmem_shrink_dcache(kmem_cache_t *, int, int);
+extern int kmem_shrink_icache(kmem_cache_t *, int, int);
+extern int kmem_shrink_dquota(kmem_cache_t *, int, int);
+
 extern int kmem_cache_shrink_nr(kmem_cache_t *);
 extern void *kmem_cache_alloc(kmem_cache_t *, int);
 extern void kmem_cache_free(kmem_cache_t *, void *);

diff -Nru a/mm/slab.c b/mm/slab.c

mm/slab.c +72 -1

@@ -213,6 +213,8 @@
 	kmem_cache_t		*slabp_cache;
 	unsigned int		growing;
 	unsigned int		dflags;		/* dynamic flags */
+	shrinker_t		shrinker;	/* shrink callback */
+	int 			count;		/* count used to trigger shrink */
 
 	/* constructor func */
 	void (*ctor)(void *, kmem_cache_t *, unsigned long);

@@ -382,6 +384,69 @@
 static void enable_cpucache (kmem_cache_t *cachep);
 static void enable_all_cpucaches (void);
 #endif
+
+/* set the shrink family and function */
+void kmem_set_shrinker(kmem_cache_t * cachep, shrinker_t theshrinker)
+{
+	cachep->shrinker = theshrinker;
+}
+
+/* used by refill_inactive_zone to determine caches that need shrinking */
+void kmem_count_page(struct page *page)
+{
+	kmem_cache_t *cachep = GET_PAGE_CACHE(page);
+	cachep->count++;
+}
+
+/* call the shrink family function */
+int kmem_call_shrinkers(int priority, int gfp_mask)
+{
+	int ret = 0;
+	struct list_head *p;
+
+        if (gfp_mask & __GFP_WAIT)
+                down(&cache_chain_sem);
+        else
+                if (down_trylock(&cache_chain_sem))
+                        return 0;
+
+        list_for_each(p,&cache_chain) {
+                kmem_cache_t *cachep = list_entry(p, kmem_cache_t, next);
+		if (cachep->count > 0) {
+			if (cachep->shrinker == NULL)
+				BUG();
+			ret += (*cachep->shrinker)(cachep, priority, gfp_mask);
+			cachep->count = 0;
+		}
+        }
+        up(&cache_chain_sem);
+	return ret;
+}
+
+/* shink methods */
+int kmem_shrink_default(kmem_cache_t * cachep, int priority, int gfp_mask)
+{
+	return kmem_cache_shrink_nr(cachep);
+}
+
+int kmem_shrink_dcache(kmem_cache_t * cachep, int priority, int gfp_mask)
+{
+	return shrink_dcache_memory(priority, gfp_mask);
+}
+
+int kmem_shrink_icache(kmem_cache_t * cachep, int priority, int gfp_mask)
+{
+	return shrink_icache_memory(priority, gfp_mask);
+}
+
+#if defined (CONFIG_QUOTA)
+
+int kmem_shrink_dquota(kmem_cache_t * cachep, int priority, int gfp_mask)
+{
+	return shrink_dqcache_memory(priority, gfp_mask);
+}
+
+#endif
 
 /* Cal the num objs, wastage, and bytes left over for a given slab size.
 */ static void kmem_cache_estimate (unsigned long gfporder, size_t size, @@
-514,6 +579,8 @@
 	 * vm_scan(). Shouldn't be a worry.
 	 */
 	while (i--) {
+		if (!(cachep->flags & SLAB_NO_REAP))
+			lru_cache_del(page);
 		PageClearSlab(page);
 		page++;
 	}

@@ -781,6 +848,8 @@
 		flags |= CFLGS_OPTIMIZE;
 
 	cachep->flags = flags;
+	cachep->shrinker = ( shrinker_t)(kmem_shrink_default);
+	cachep->count = 0;
 	cachep->gfpflags = 0;
 	if (flags & SLAB_CACHE_DMA)
 		cachep->gfpflags |= GFP_DMA;

@@ -1184,6 +1253,8 @@
 		SET_PAGE_CACHE(page, cachep);
 		SET_PAGE_SLAB(page, slabp);
 		PageSetSlab(page);
+		if (!(cachep->flags & SLAB_NO_REAP))
+			lru_cache_add(page);
 		page++;
 	} while (--i);
 

@@ -1903,6 +1974,7 @@
 		unsigned long	num_objs;
 		unsigned long	active_slabs = 0;
 		unsigned long	num_slabs;
+		int		ref;
 		cachep = list_entry(p, kmem_cache_t, next);
 
 		spin_lock_irq(&cachep->spinlock);

diff -Nru a/mm/vmscan.c b/mm/vmscan.c

mm/vmscan.c +19 -3

@@ -102,6 +102,9 @@
 			continue;
 		}
 
+		if (PageSlab(page))
+			BUG();
+
 		/* Page is being freed */
 		if (unlikely(page_count(page)) == 0) {
 			list_del(page_lru);

@@ -244,7 +247,8 @@
 		 * The page is in active use or really unfreeable. Move to
 		 * the active list and adjust the page age if needed.
 		 */
-		if (page_referenced(page) && page_mapping_inuse(page) &&
+		if (page_referenced(page) &&
+				(page_mapping_inuse(page) || PageSlab(page)) &&
 				!page_over_rsslimit(page)) {
 			del_page_from_inactive_dirty_list(page);
 			add_page_to_active_list(page);

@@ -253,6 +257,12 @@
 		}
 
 		/*
+		 * SlabPages get shrunk in refill_inactive_zone
+		 */
+		if (PageSlab(page))
+			continue;
+
+		/*
 		 * Page is being freed, don't worry about it.
 		 */
 		if (unlikely(page_count(page)) == 0)

@@ -446,6 +456,7 @@
  * This function will scan a portion of the active list of a zone to find
  * unused pages, those pages will then be moved to the inactive list.
  */
+
 int refill_inactive_zone(struct zone_struct * zone, int priority)
 {
 	int maxscan = zone->active_pages >> priority;

@@ -473,7 +484,7 @@
 		 * bother with page aging.  If the page is touched again
 		 * while on the inactive_clean list it'll be reactivated.
 		 */
-		if (!page_mapping_inuse(page)) {
+		if (!page_mapping_inuse(page) && !PageSlab(page)) {
 			drop_page(page);
 			continue;
 		}

@@ -497,8 +508,12 @@
 			list_add(page_lru, &zone->active_list);
 		} else {
 			deactivate_page_nolock(page);
-			if (++nr_deactivated > target)
+			if (PageSlab(page))
+				kmem_count_page(page);
+			else {
+				if (++nr_deactivated > target)
 				break;
+			}
 		}
 
 		/* Low latency reschedule point */

@@ -513,6 +528,7 @@
 	return nr_deactivated;
 }
 
+
 /**
  * refill_inactive - checks all zones and refills the inactive list as

needed *

mm/vmscan.c +9 -15

@@ -577,24 +593,15 @@
 
 	/*
 	 * Eat memory from filesystem page cache, buffer cache,
-	 * dentry, inode and filesystem quota caches.
 	 */
 	ret += page_launder(gfp_mask);
-	ret += shrink_dcache_memory(DEF_PRIORITY, gfp_mask);
-	ret += shrink_icache_memory(1, gfp_mask);
-#ifdef CONFIG_QUOTA
-	ret += shrink_dqcache_memory(DEF_PRIORITY, gfp_mask);
-#endif
 
 	/*
-	 * Move pages from the active list to the inactive list.
+	 * Move pages from the active list to the inactive list and
+	 * shrink caches return pages gained by shrink
 	 */
 	refill_inactive();
-
-	/*
-	 * Reclaim unused slab cache memory.
-	 */
-	ret += kmem_cache_reap(gfp_mask);
+	ret += kmem_call_shrinkers(DEF_PRIORITY, gfp_mask);
 
 	refill_freelist();
 

@@ -603,11 +610,14 @@
 		run_task_queue(&tq_disk);
 
 	/*
-	 * Hmm.. Cache shrink failed - time to kill something?
+	 * Hmm.. - time to kill something?
 	 * Mhwahahhaha! This is the part I really like. Giggle.
 	 */
-	if (!ret && free_min(ANY_ZONE) > 0)
-		out_of_memory();
+	if (!ret && free_min(ANY_ZONE) > 0) {
+		ret += kmem_cache_reap(gfp_mask);
+		if (!ret)
+			out_of_memory();
+	}
 
 	return ret;
 }

Quoted 25 lines

When running under low vm pressure rmap does not shrink caches. This
happens since we only call do_try_to_free_pages when we have a shortage. On my box the combination of background_aging calling refill_inactive_zone is able to supply the pages needed. The end result of this the box acts sluggish, with about half my memory used by slab pages (dcache/icache). This does correct itself under pressure but it should never get into this state in the first place.

Idealy we want all pages to be about the same age. Having half the pages in the system 'cold' in the slab cache is not good - it implies the other pages are 'hotter' than they need to be.

To fix the situation I move reapable slab pages into the active list. When aging moves a page into the inactive dirty list I watch for slab pages and record the caches with old pages. After refill_inactive/background_aging ends I call a new function, kmem_call_shrinkers. This scans the list of slab caches and, via a callback, shrinks caches with old pages. Note that we never swap out slab pages they just cycle through active and inactive dirty lists.

The end result is that slab caches are shrunk selectivily when they have old 'cold' pages. I avoid adding any magic numbers to the vm and create a generic interface to allow creators of slab caches to supply the vm with a unique method to shrink their caches.

Quoted 4 lines

When testing this there is one side effect to remember. Using cat /proc/slabinfo references pages - this will tend to keep the slab pages warmer than they should be. Like in quantum theory, watching (to often) can change results.

One additional comment. I have tried modifing kmem_cache_shrink_nr to free only the number of pages seen by refill_inactive_zone. This scheme revives the original problem. Think the issue is that, in essence, the the dentry/inode caches often work in read once mode (thats each object in
a slab is used once...). Without the more aggresive shrink in this patch
the 'read once' slab pages upset the vm balance.

A data point. Comparing this patch to my previous one the inode/entry
caches stabilize at about twice the size here.

Quoted 1 line

I have testing on UP only - think the locking is ok though...

Andrew Morton pointed out that the kernel is using 8m pages and is setting reference bits for these pages... He suggested (amoung other things - thanks) that setting the bits in kmem_cache_alloc would be
a good start to making aging happen. This version of the patch
impliments his suggestion.
Comments?
Ed Tomlinson