Containers: Pagecache accounting and control subsystem (v1)

-----------------------------------------------------------

This patch adds pagecache accounting and control on top of Paul's container subsystem v7 posted at
http://lkml.org/lkml/2007/2/12/88

and Balbir's RSS controller posted at
http://lkml.org/lkml/2007/2/26/8

This patchset depends on Balbir's RSS controller and cannot work independent of it. The page reclaim code has been merged with container RSS controller. However compile time options can individually enable/disable memory controller and/or pagecache controller.

Comments, suggestions and criticisms are welcome.

Features:
--------
* New subsystem called 'pagecache_acct' is registered with containers
* Container pointer is added to struct address_space to keep track of associated container
* In filemap.c and swap_state.c, the corresponding container's pagecache_acct subsystem is charged and uncharged whenever a new page is added or removed from pagecache
* The accounting number include pages in swap cache and filesystem buffer pages apart from pagecache, basically everything under NR_FILE_PAGES is counted as pagecache. However this excluded mapped and anonymous pages
* Limits on pagecache can be set by echo 100000 > pagecache_limit on the /container file system. The unit is in kilobytes
* If the pagecache utilisation limit is exceeded, pagecache reclaim code is invoked to recover dirty and clean pagecache pages only.

Advantages:
-----------
* Does not add container pointers in struct page

Limitations:
-----------
* Code is not safe for container deletion/task migration
* Pagecache page reclaim needs performance improvements
* Global LRU is churned in search of pagecache pages

Usage:
-----

* Add patch on top of Paul container (v7) at kernel version 2.6.20
* Enable CONFIG_CONTAINER_PAGECACHE_ACCT in 'General Setup'
* Boot new kernel
* Mount container filesystem
mount -t container /container
cd /container
* Create new container
mkdir mybox
cd /container/mybox
* Add current shell to container
echo $$ > tasks
* There are two files pagecache_usage and pagecache_limit
* In order to set limit, echo value in kilobytes to pagecache_limit

	echo 100000 > pagecache_limit 
	#This would set 100MB limit on pagecache usage

* Trash the system from current shell using scp/cp/dd/tar etc
* Watch pagecache_usage and /proc/meminfo to verify behavior

* Only unmapped pagecache data will be accounted and controlled. These are memory used by cp, scp, tar etc. While file mmap will be controlled by Balbir's RSS controller.

Tests:
------

* Ran kernbench within container with pagecache_limits set

ToDo:
----

* Merge with container RSS controller and eliminate redundant code
* Test and support task migration and container deletion
* Review reclaim performance
* Optimise page reclaim

Patch Series:
-------------
pagecache-controller-setup.patch
pagecache-controller-acct.patch
pagecache-controller-reclaim.patch

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

This patch adds basic infrastructure for pagecache accounting and control subsystem within the container framework.

pagecache_usage and pagecache_limit files are created along with every new container. Routines to extract subsystem pointers and container pointers have been added.

Compile time kernel options and have been added along with Makefile changes.

---
 include/linux/pagecache_acct.h |   53 +++++
 init/Kconfig                   |    7 
 mm/Makefile                    |    1 
 mm/pagecache_acct.c            |  368 +++++++++++++++++++++++++++++++++++++++++
 4 files changed, 429 insertions(+)

@@ -0,0 +1,53 @@
+/*
+ * Pagecache controller - "Account and control pagecache usage"
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.
+ *
+ * Copyright IBM Corporation, 2007
+ *
+ * Author: Vaidyanathan Srinivasan <[email protected]>
+ *
+ */
+
+#ifndef _LINUX_PAGECACHE_ACCT_H
+#define _LINUX_PAGECACHE_ACCT_H
+
+#include <linux/container.h>
+
+#ifdef CONFIG_CONTAINER_PAGECACHE_ACCT
+extern void pagecache_acct_init_page_ptr(struct page *page);
+extern void pagecache_acct_charge(struct page *page);
+extern void pagecache_acct_uncharge(struct page *page);
+extern int pagecache_acct_page_overlimit(struct page *page);
+extern int pagecache_acct_mapping_overlimit(struct address_space *mapping);
+extern int pagecache_acct_cont_overlimit(struct container *cont);
+extern int pagecache_acct_shrink_used(unsigned long nr_pages);
+#else
+static inline void pagecache_acct_init_page_ptr(struct page *page) {}
+static inline void pagecache_acct_charge(struct page *page) {}
+static inline void pagecache_acct_uncharge(struct page *page) {}
+static inline int pagecache_acct_page_overlimit(
+		struct page *page) { return 0; }
+static inline int pagecache_acct_mapping_overlimit(
+		struct address_space *mapping) { return 0; }
+static inline int pagecache_acct_cont_overlimit(
+		struct container *cont) { return 0; }
+static inline int pagecache_acct_shrink_used(
+		unsigned long nr_pages) { return 0; }
+#endif /* CONFIG_CONTAINER_PAGECACHE_ACCT */
+
+#endif /* _LINUX_PAGECACHE_ACCT_H */
+
+

@@ -313,6 +313,13 @@ config CONTAINER_MEMCONTROL
 	  Provides a simple Resource Controller for monitoring and
 	  controlling the total Resident Set Size of the tasks in a container
 
+config CONTAINER_PAGECACHE_ACCT
+	bool "Simple PageCache accounting & control container subsystem"
+	select CONTAINERS
+	help
+	  Provides a simple Resource Controller for monitoring the
+	  total pagecache memory consumed by the tasks in a container
+
 config RELAY
 	bool "Kernel->user space relay support (formerly relayfs)"
 	help

@@ -30,3 +30,4 @@ obj-$(CONFIG_FS_XIP) += filemap_xip.o
 obj-$(CONFIG_MIGRATION) += migrate.o
 obj-$(CONFIG_SMP) += allocpercpu.o
 obj-$(CONFIG_CONTAINER_MEMCONTROL) += memcontrol.o
+obj-$(CONFIG_CONTAINER_PAGECACHE_ACCT) += pagecache_acct.o

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

The accounting framework works by adding a container pointer in address_space structure. Each page in pagecache belongs to a radix tree within the address_space structure corresponding to the inode.

In order to charge the container for pagecache usage, the corresponding address_space is obtained from struct page which holds the container pointer. This framework avoids any additional pointers in struct page.

additions and deletions from pagecache are hooked to charge and uncharge the corresponding container.

---
 include/linux/fs.h |    4 ++++
 mm/filemap.c       |    8 ++++++++
 2 files changed, 12 insertions(+)

@@ -447,6 +447,10 @@ struct address_space {
 	spinlock_t		private_lock;	/* for use by the address_space */
 	struct list_head	private_list;	/* ditto */
 	struct address_space	*assoc_mapping;	/* ditto */
+#ifdef CONFIG_CONTAINER_PAGECACHE_ACCT
+	struct container *container; 	/* Charge page to the right container
+					   using page->mapping */
+#endif
 } __attribute__((aligned(sizeof(long))));
 	/*
 	 * On most architectures that alignment is already the case; but

@@ -30,6 +30,7 @@
 #include <linux/security.h>
 #include <linux/syscalls.h>
 #include <linux/cpuset.h>
+#include <linux/pagecache_acct.h>
 #include "filemap.h"
 #include "internal.h"
 

@@ -117,6 +118,8 @@ void __remove_from_page_cache(struct pag
 	struct address_space *mapping = page->mapping;
 
 	radix_tree_delete(&mapping->page_tree, page->index);
+	/* Uncharge before the mapping is gone */
+	pagecache_acct_uncharge(page);
 	page->mapping = NULL;
 	mapping->nrpages--;
 	__dec_zone_page_state(page, NR_FILE_PAGES);

@@ -451,6 +454,11 @@ int add_to_page_cache(struct page *page,
 			__inc_zone_page_state(page, NR_FILE_PAGES);
 		}
 		write_unlock_irq(&mapping->tree_lock);
+		/* Unlock before charge, because we may reclaim this inline */
+		if (!error) {
+			pagecache_acct_init_page_ptr(page);
+			pagecache_acct_charge(page);
+		}
 		radix_tree_preload_end();
 	}
 	return error;

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

The reclaim code is similar to RSS memory controller. Scan control is slightly different since we are targeting different type of pages.

Additionally no mapped pages are touched when scanning for pagecache pages.

RSS memory controller and pagecache controller share common code in reclaim and hence pagecache controller patches are dependent on RSS memory controller patch even though the features are independently configurable at compile time.

---
 include/linux/memcontrol.h |    6 +++
 mm/memcontrol.c            |    4 +-
 mm/pagecache_acct.c        |   11 ++++++
 mm/vmscan.c                |   71 +++++++++++++++++++++++++++++++++++++--------
 4 files changed, 77 insertions(+), 15 deletions(-)

@@ -36,6 +36,12 @@ enum {
 	MEMCONTROL_DONT_CHECK_LIMIT = false,
 };
 
+/* Type of memory to reclaim in shrink_container_memory() */
+enum {
+	RECLAIM_MAPPED_MEMORY = 1,
+	RECLAIM_PAGECACHE_MEMORY,
+};
+
 #ifdef CONFIG_CONTAINER_MEMCONTROL
 #include <linux/wait.h>
 

@@ -165,8 +165,8 @@ static int memcontrol_check_and_reclaim(
 				nr_pages = (pushback * limit) / 100;
 			mem->reclaim_in_progress = true;
 			spin_unlock(&mem->lock);
-			nr_reclaimed +=
-				memcontrol_shrink_mapped_memory(nr_pages, cont);
+			nr_reclaimed += shrink_container_memory(
+					RECLAIM_MAPPED_MEMORY, nr_pages, cont);
 			spin_lock(&mem->lock);
 			mem->reclaim_in_progress = false;
 			wake_up_all(&mem->wq);

@@ -29,6 +29,7 @@
 #include <linux/uaccess.h>
 #include <asm/div64.h>
 #include <linux/pagecache_acct.h>
+#include <linux/memcontrol.h>
 
 /*
  * Convert unit from pages to kilobytes

@@ -337,12 +338,20 @@ int pagecache_acct_cont_overlimit(struct
 		return 0;
 }
 
-extern unsigned long shrink_all_pagecache_memory(unsigned long nr_pages);
+extern unsigned long shrink_container_memory(unsigned int memory_type,
+				unsigned long nr_pages, void *container);
 
 int pagecache_acct_shrink_used(unsigned long nr_pages)
 {
 	unsigned long ret = 0;
 	atomic_inc(&reclaim_count);
+
+	/* Don't call reclaim for each page above limit */
+	if (nr_pages > NR_PAGES_RECLAIM_THRESHOLD) {
+		ret += shrink_container_memory(
+				RECLAIM_PAGECACHE_MEMORY, nr_pages, NULL);
+	}
+
 	return 0;
 }
 

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Anyone else have trouble fitting this patch ???.
I see a later version today, but not markedly different from this mornings (Aus time). Initially I thought I had the first version, prior to Balbir's RSS controller V2 re-write, but apparently not. Kernel 2.6.20.1

Had to toss it away so I could do some base line testing - I'll redo the build and see where the mis-matches are.

Shane ...

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Hi Shane,

I did post the same patch again today since the previous post yesterday did not showup on LKML. I have not changed the version since it is the same patch.

Next time around i will explicitly mention that this is the same patch posted again.

Please let me know if so see any problem running the patch. The patches are against 2.6.20 only since dependent patches are at that level.

--Vaidy

My problem - a bad copy of the patch. It patches o.k. However, it fails to compile vmscan. This looks a bit dodgy;

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

@@ -1470,11 +1494,13 @@ unsigned long shrink_all_memory(unsigned

        int pass;
        struct reclaim_state reclaim_state;
        struct scan_control sc = {

- .gfp_mask = GFP_KERNEL,
+ .gfp_mask = GFdefined(CONFIG_CONTAINER_PAGECACHE_ACCT)
+P_KERNEL,

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

I deleted what looks like an over-enthusiastic "copy-and-paste", and it compiled o.k.
Testing continues.

Shane ...

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

OOPS!! How did it get there! That is certainly some random mouse click. Thanks for pointing that out.

--Vaidy