Memory controller soft limit patches (v6)

Feature: Organize cgroups over soft limit in a RB-Tree

From: Balbir Singh <[email protected]>

Changelog v6...v5
1. Update the key before inserting into RB tree. Without the current change it could take an additional iteration to get the key correct.

Changelog v5...v4
1. res_counter_uncharge has an additional parameter to indicate if the counter was over its soft limit, before uncharge.

Changelog v4...v3
1. Optimizations to ensure we don't uncessarily get res_counter values
2. Fixed a bug in usage of time_after()

Changelog v3...v2
1. Add only the ancestor to the RB-Tree
2. Use css_tryget/css_put instead of mem_cgroup_get/mem_cgroup_put

Changelog v2...v1
1. Add support for hierarchies
2. The res_counter that is highest in the hierarchy is returned on soft limit being exceeded. Since we do hierarchical reclaim and add all groups exceeding their soft limits, this approach seems to work well in practice.

This patch introduces a RB-Tree for storing memory cgroups that are over their soft limit. The overall goal is to

1. Add a memory cgroup to the RB-Tree when the soft limit is exceeded. We are careful about updates, updates take place only after a particular time interval has passed
2. We remove the node from the RB-Tree when the usage goes below the soft limit

The next set of patches will exploit the RB-Tree to get the group that is over its soft limit by the largest amount and reclaim from it, when we face memory contention.

Signed-off-by Balbir Singh <[email protected]>

---

 include/linux/res_counter.h |    6 +-
 kernel/res_counter.c        |   18 +++++
 mm/memcontrol.c             |  141 ++++++++++++++++++++++++++++++++++++++-----
 3 files changed, 143 insertions(+), 22 deletions(-)

include/linux/res_counter.h +4 -2

index 5c821fd..5bbf8b1 100644
@@ -112,7 +112,8 @@ void res_counter_init(struct res_counter *counter, struct res_counter *parent);
 int __must_check res_counter_charge_locked(struct res_counter *counter,
 		unsigned long val);
 int __must_check res_counter_charge(struct res_counter *counter,
-		unsigned long val, struct res_counter **limit_fail_at);
+		unsigned long val, struct res_counter **limit_fail_at,
+		struct res_counter **soft_limit_at);
 
 /*
  * uncharge - tell that some portion of the resource is released

@@ -125,7 +126,8 @@ int __must_check res_counter_charge(struct res_counter *counter,
  */
 
 void res_counter_uncharge_locked(struct res_counter *counter, unsigned long val);
-void res_counter_uncharge(struct res_counter *counter, unsigned long val);
+void res_counter_uncharge(struct res_counter *counter, unsigned long val,
+				bool *was_soft_limit_excess);
 
 static inline bool res_counter_limit_check_locked(struct res_counter *cnt)
 {

kernel/res_counter.c +16 -2

index 4e6dafe..51ec438 100644
@@ -37,17 +37,27 @@ int res_counter_charge_locked(struct res_counter *counter, unsigned long val)
 }
 
 int res_counter_charge(struct res_counter *counter, unsigned long val,
-			struct res_counter **limit_fail_at)
+			struct res_counter **limit_fail_at,
+			struct res_counter **soft_limit_fail_at)
 {
 	int ret;
 	unsigned long flags;
 	struct res_counter *c, *u;
 
 	*limit_fail_at = NULL;
+	if (soft_limit_fail_at)
+		*soft_limit_fail_at = NULL;
 	local_irq_save(flags);
 	for (c = counter; c != NULL; c = c->parent) {
 		spin_lock(&c->lock);
 		ret = res_counter_charge_locked(c, val);
+		/*
+		 * With soft limits, we return the highest ancestor
+		 * that exceeds its soft limit
+		 */
+		if (soft_limit_fail_at &&
+			!res_counter_soft_limit_check_locked(c))
+			*soft_limit_fail_at = c;
 		spin_unlock(&c->lock);
 		if (ret < 0) {
 			*limit_fail_at = c;

@@ -75,7 +85,8 @@ void res_counter_uncharge_locked(struct res_counter *counter, unsigned long val)
 	counter->usage -= val;
 }
 
-void res_counter_uncharge(struct res_counter *counter, unsigned long val)
+void res_counter_uncharge(struct res_counter *counter, unsigned long val,
+				bool *was_soft_limit_excess)
 {
 	unsigned long flags;
 	struct res_counter *c;

@@ -83,6 +94,9 @@ void res_counter_uncharge(struct res_counter *counter, unsigned long val)
 	local_irq_save(flags);
 	for (c = counter; c != NULL; c = c->parent) {
 		spin_lock(&c->lock);
+		if (c == counter && was_soft_limit_excess)
+			*was_soft_limit_excess =
+				!res_counter_soft_limit_check_locked(c);
 		res_counter_uncharge_locked(c, val);
 		spin_unlock(&c->lock);
 	}

mm/memcontrol.c +88 -2

index 70bc992..200d44a 100644
@@ -29,6 +29,7 @@
 #include <linux/rcupdate.h>
 #include <linux/limits.h>
 #include <linux/mutex.h>
+#include <linux/rbtree.h>
 #include <linux/slab.h>
 #include <linux/swap.h>
 #include <linux/spinlock.h>

@@ -129,6 +130,14 @@ struct mem_cgroup_lru_info {
 };
 
 /*
+ * Cgroups above their limits are maintained in a RB-Tree, independent of
+ * their hierarchy representation
+ */
+
+static struct rb_root mem_cgroup_soft_limit_tree;
+static DEFINE_SPINLOCK(memcg_soft_limit_tree_lock);
+
+/*
  * The memory controller data structure. The memory controller controls both
  * page cache and RSS per cgroup. We would eventually like to provide
  * statistics based on the statistics developed by Rik Van Riel for clock-pro,

@@ -176,12 +185,20 @@ struct mem_cgroup {
 
 	unsigned int	swappiness;
 
+	struct rb_node mem_cgroup_node;		/* RB tree node */
+	unsigned long long usage_in_excess;	/* Set to the value by which */
+						/* the soft limit is exceeded*/
+	unsigned long last_tree_update;		/* Last time the tree was */
+						/* updated in jiffies     */
+
 	/*
 	 * statistics. This must be placed at the end of memcg.
 	 */
 	struct mem_cgroup_stat stat;
 };
 
+#define	MEM_CGROUP_TREE_UPDATE_INTERVAL		(HZ/4)
+
 enum charge_type {
 	MEM_CGROUP_CHARGE_TYPE_CACHE = 0,
 	MEM_CGROUP_CHARGE_TYPE_MAPPED,

@@ -214,6 +231,42 @@ static void mem_cgroup_get(struct mem_cgroup *mem);
 static void mem_cgroup_put(struct mem_cgroup *mem);
 static struct mem_cgroup *parent_mem_cgroup(struct mem_cgroup *mem);
 
+static void mem_cgroup_insert_exceeded(struct mem_cgroup *mem)
+{
+	struct rb_node **p = &mem_cgroup_soft_limit_tree.rb_node;
+	struct rb_node *parent = NULL;
+	struct mem_cgroup *mem_node;
+	unsigned long flags;
+
+	spin_lock_irqsave(&memcg_soft_limit_tree_lock, flags);
+	mem->usage_in_excess = res_counter_soft_limit_excess(&mem->res);
+	while (*p) {
+		parent = *p;
+		mem_node = rb_entry(parent, struct mem_cgroup, mem_cgroup_node);
+		if (mem->usage_in_excess < mem_node->usage_in_excess)
+			p = &(*p)->rb_left;
+		/*
+		 * We can't avoid mem cgroups that are over their soft
+		 * limit by the same amount
+		 */
+		else if (mem->usage_in_excess >= mem_node->usage_in_excess)
+			p = &(*p)->rb_right;
+	}
+	rb_link_node(&mem->mem_cgroup_node, parent, p);
+	rb_insert_color(&mem->mem_cgroup_node,
+			&mem_cgroup_soft_limit_tree);
+	mem->last_tree_update = jiffies;
+	spin_unlock_irqrestore(&memcg_soft_limit_tree_lock, flags);
+}
+
+static void mem_cgroup_remove_exceeded(struct mem_cgroup *mem)
+{
+	unsigned long flags;
+	spin_lock_irqsave(&memcg_soft_limit_tree_lock, flags);
+	rb_erase(&mem->mem_cgroup_node, &mem_cgroup_soft_limit_tree);
+	spin_unlock_irqrestore(&memcg_soft_limit_tree_lock, flags);
+}
+
 static void mem_cgroup_charge_statistics(struct mem_cgroup *mem,
 					 struct page_cgroup *pc,
 					 bool charge)

@@ -897,6 +950,39 @@ static void record_last_oom(struct mem_cgroup *mem)
 	mem_cgroup_walk_tree(mem, NULL, record_last_oom_cb);
 }
 
+static void mem_cgroup_check_and_update_tree(struct mem_cgroup *mem,
+						bool time_check)
+{
+	unsigned long long prev_usage_in_excess, new_usage_in_excess;
+	bool updated_tree = false;
+	unsigned long next_update = 0;
+	unsigned long flags;
+
+	prev_usage_in_excess = mem->usage_in_excess;
+
+	if (time_check)
+		next_update = mem->last_tree_update +
+				MEM_CGROUP_TREE_UPDATE_INTERVAL;
+
+	if (!time_check || time_after(jiffies, next_update)) {
+		new_usage_in_excess = res_counter_soft_limit_excess(&mem->res);
+		if (prev_usage_in_excess) {
+			mem_cgroup_remove_exceeded(mem);
+			updated_tree = true;
+		}
+		if (!new_usage_in_excess)
+			goto done;
+		mem_cgroup_insert_exceeded(mem);
+	}
+
+done:
+	if (updated_tree) {
+		spin_lock_irqsave(&memcg_soft_limit_tree_lock, flags);
+		mem->last_tree_update = jiffies;
+		mem->usage_in_excess = new_usage_in_excess;
+		spin_unlock_irqrestore(&memcg_soft_limit_tree_lock, flags);
+	}
+}
 
 /*
  * Unlike exported interface, "oom" parameter is added. if oom==true,

@@ -906,9 +992,9 @@ static int __mem_cgroup_try_charge(struct mm_struct *mm,
 			gfp_t gfp_mask, struct mem_cgroup **memcg,
 			bool oom)
 {
-	struct mem_cgroup *mem, *mem_over_limit;
+	struct mem_cgroup *mem, *mem_over_limit, *mem_over_soft_limit;
 	int nr_retries = MEM_CGROUP_RECLAIM_RETRIES;
-	struct res_counter *fail_res;
+	struct res_counter *fail_res, *soft_fail_res = NULL;
 
 	if (unlikely(test_thread_flag(TIF_MEMDIE))) {
 		/* Don't account this! */

Quoted 270 lines · On Sat, 14 Mar 2009 23:01:02 +0530 Balbir Singh <[email protected]>…

Feature: Organize cgroups over soft limit in a RB-Tree

From: Balbir Singh <[email protected]>

Changelog v6...v5
1. Update the key before inserting into RB tree. Without the current change it could take an additional iteration to get the key correct.

Changelog v5...v4
1. res_counter_uncharge has an additional parameter to indicate if the counter was over its soft limit, before uncharge.

Changelog v4...v3
1. Optimizations to ensure we don't uncessarily get res_counter values
2. Fixed a bug in usage of time_after()

Changelog v3...v2
1. Add only the ancestor to the RB-Tree
2. Use css_tryget/css_put instead of mem_cgroup_get/mem_cgroup_put

Changelog v2...v1
1. Add support for hierarchies
2. The res_counter that is highest in the hierarchy is returned on soft limit being exceeded. Since we do hierarchical reclaim and add all groups exceeding their soft limits, this approach seems to work well in practice.

This patch introduces a RB-Tree for storing memory cgroups that are over their soft limit. The overall goal is to

1. Add a memory cgroup to the RB-Tree when the soft limit is exceeded. We are careful about updates, updates take place only after a particular time interval has passed
2. We remove the node from the RB-Tree when the usage goes below the soft limit

The next set of patches will exploit the RB-Tree to get the group that is over its soft limit by the largest amount and reclaim from it, when we face memory contention.

Signed-off-by Balbir Singh <[email protected]>

---

 include/linux/res_counter.h |    6 +-
 kernel/res_counter.c        |   18 +++++
 mm/memcontrol.c             |  141 ++++++++++++++++++++++++++++++++++++++-----
 3 files changed, 143 insertions(+), 22 deletions(-)

include/linux/res_counter.h +4 -2

index 5c821fd..5bbf8b1 100644
@@ -112,7 +112,8 @@ void res_counter_init(struct res_counter *counter, struct res_counter *parent);
 int __must_check res_counter_charge_locked(struct res_counter *counter,
 		unsigned long val);
 int __must_check res_counter_charge(struct res_counter *counter,
-		unsigned long val, struct res_counter **limit_fail_at);
+		unsigned long val, struct res_counter **limit_fail_at,
+		struct res_counter **soft_limit_at);
 
 /*
  * uncharge - tell that some portion of the resource is released

@@ -125,7 +126,8 @@ int __must_check res_counter_charge(struct res_counter *counter,
  */
 
 void res_counter_uncharge_locked(struct res_counter *counter, unsigned long val);
-void res_counter_uncharge(struct res_counter *counter, unsigned long val);
+void res_counter_uncharge(struct res_counter *counter, unsigned long val,
+				bool *was_soft_limit_excess);
 
 static inline bool res_counter_limit_check_locked(struct res_counter *cnt)
 {

kernel/res_counter.c +16 -2

index 4e6dafe..51ec438 100644
@@ -37,17 +37,27 @@ int res_counter_charge_locked(struct res_counter *counter, unsigned long val)
 }
 
 int res_counter_charge(struct res_counter *counter, unsigned long val,
-			struct res_counter **limit_fail_at)
+			struct res_counter **limit_fail_at,
+			struct res_counter **soft_limit_fail_at)
 {
 	int ret;
 	unsigned long flags;
 	struct res_counter *c, *u;
 
 	*limit_fail_at = NULL;
+	if (soft_limit_fail_at)
+		*soft_limit_fail_at = NULL;
 	local_irq_save(flags);
 	for (c = counter; c != NULL; c = c->parent) {
 		spin_lock(&c->lock);
 		ret = res_counter_charge_locked(c, val);
+		/*
+		 * With soft limits, we return the highest ancestor
+		 * that exceeds its soft limit
+		 */
+		if (soft_limit_fail_at &&
+			!res_counter_soft_limit_check_locked(c))
+			*soft_limit_fail_at = c;
 		spin_unlock(&c->lock);
 		if (ret < 0) {
 			*limit_fail_at = c;

@@ -75,7 +85,8 @@ void res_counter_uncharge_locked(struct res_counter *counter, unsigned long val)
 	counter->usage -= val;
 }
 
-void res_counter_uncharge(struct res_counter *counter, unsigned long val)
+void res_counter_uncharge(struct res_counter *counter, unsigned long val,
+				bool *was_soft_limit_excess)
 {
 	unsigned long flags;
 	struct res_counter *c;

@@ -83,6 +94,9 @@ void res_counter_uncharge(struct res_counter *counter, unsigned long val)
 	local_irq_save(flags);
 	for (c = counter; c != NULL; c = c->parent) {
 		spin_lock(&c->lock);
+		if (c == counter && was_soft_limit_excess)
+			*was_soft_limit_excess =
+				!res_counter_soft_limit_check_locked(c);
 		res_counter_uncharge_locked(c, val);
 		spin_unlock(&c->lock);
 	}

mm/memcontrol.c +88 -2

index 70bc992..200d44a 100644
@@ -29,6 +29,7 @@
 #include <linux/rcupdate.h>
 #include <linux/limits.h>
 #include <linux/mutex.h>
+#include <linux/rbtree.h>
 #include <linux/slab.h>
 #include <linux/swap.h>
 #include <linux/spinlock.h>

@@ -129,6 +130,14 @@ struct mem_cgroup_lru_info {
 };
 
 /*
+ * Cgroups above their limits are maintained in a RB-Tree, independent of
+ * their hierarchy representation
+ */
+
+static struct rb_root mem_cgroup_soft_limit_tree;
+static DEFINE_SPINLOCK(memcg_soft_limit_tree_lock);
+
+/*
  * The memory controller data structure. The memory controller controls both
  * page cache and RSS per cgroup. We would eventually like to provide
  * statistics based on the statistics developed by Rik Van Riel for clock-pro,

@@ -176,12 +185,20 @@ struct mem_cgroup {
 
 	unsigned int	swappiness;
 
+	struct rb_node mem_cgroup_node;		/* RB tree node */
+	unsigned long long usage_in_excess;	/* Set to the value by which */
+						/* the soft limit is exceeded*/
+	unsigned long last_tree_update;		/* Last time the tree was */
+						/* updated in jiffies     */
+
 	/*
 	 * statistics. This must be placed at the end of memcg.
 	 */
 	struct mem_cgroup_stat stat;
 };
 
+#define	MEM_CGROUP_TREE_UPDATE_INTERVAL		(HZ/4)
+
 enum charge_type {
 	MEM_CGROUP_CHARGE_TYPE_CACHE = 0,
 	MEM_CGROUP_CHARGE_TYPE_MAPPED,

@@ -214,6 +231,42 @@ static void mem_cgroup_get(struct mem_cgroup *mem);
 static void mem_cgroup_put(struct mem_cgroup *mem);
 static struct mem_cgroup *parent_mem_cgroup(struct mem_cgroup *mem);
 
+static void mem_cgroup_insert_exceeded(struct mem_cgroup *mem)
+{
+	struct rb_node **p = &mem_cgroup_soft_limit_tree.rb_node;
+	struct rb_node *parent = NULL;
+	struct mem_cgroup *mem_node;
+	unsigned long flags;
+
+	spin_lock_irqsave(&memcg_soft_limit_tree_lock, flags);
+	mem->usage_in_excess = res_counter_soft_limit_excess(&mem->res);
+	while (*p) {
+		parent = *p;
+		mem_node = rb_entry(parent, struct mem_cgroup, mem_cgroup_node);
+		if (mem->usage_in_excess < mem_node->usage_in_excess)
+			p = &(*p)->rb_left;
+		/*
+		 * We can't avoid mem cgroups that are over their soft
+		 * limit by the same amount
+		 */
+		else if (mem->usage_in_excess >= mem_node->usage_in_excess)
+			p = &(*p)->rb_right;
+	}
+	rb_link_node(&mem->mem_cgroup_node, parent, p);
+	rb_insert_color(&mem->mem_cgroup_node,
+			&mem_cgroup_soft_limit_tree);
+	mem->last_tree_update = jiffies;
+	spin_unlock_irqrestore(&memcg_soft_limit_tree_lock, flags);
+}
+
+static void mem_cgroup_remove_exceeded(struct mem_cgroup *mem)
+{
+	unsigned long flags;
+	spin_lock_irqsave(&memcg_soft_limit_tree_lock, flags);
+	rb_erase(&mem->mem_cgroup_node, &mem_cgroup_soft_limit_tree);
+	spin_unlock_irqrestore(&memcg_soft_limit_tree_lock, flags);
+}
+
 static void mem_cgroup_charge_statistics(struct mem_cgroup *mem,
 					 struct page_cgroup *pc,
 					 bool charge)

@@ -897,6 +950,39 @@ static void record_last_oom(struct mem_cgroup *mem)
 	mem_cgroup_walk_tree(mem, NULL, record_last_oom_cb);
 }
 
+static void mem_cgroup_check_and_update_tree(struct mem_cgroup *mem,
+						bool time_check)
+{
+	unsigned long long prev_usage_in_excess, new_usage_in_excess;
+	bool updated_tree = false;
+	unsigned long next_update = 0;
+	unsigned long flags;
+
+	prev_usage_in_excess = mem->usage_in_excess;
+
+	if (time_check)
+		next_update = mem->last_tree_update +
+				MEM_CGROUP_TREE_UPDATE_INTERVAL;
+
+	if (!time_check || time_after(jiffies, next_update)) {
+		new_usage_in_excess = res_counter_soft_limit_excess(&mem->res);
+		if (prev_usage_in_excess) {
+			mem_cgroup_remove_exceeded(mem);
+			updated_tree = true;
+		}
+		if (!new_usage_in_excess)
+			goto done;
+		mem_cgroup_insert_exceeded(mem);
+	}
+
+done:
+	if (updated_tree) {
+		spin_lock_irqsave(&memcg_soft_limit_tree_lock, flags);
+		mem->last_tree_update = jiffies;
+		mem->usage_in_excess = new_usage_in_excess;
+		spin_unlock_irqrestore(&memcg_soft_limit_tree_lock, flags);
+	}
+}
 
 /*
  * Unlike exported interface, "oom" parameter is added. if oom==true,

@@ -906,9 +992,9 @@ static int __mem_cgroup_try_charge(struct mm_struct *mm,
 			gfp_t gfp_mask, struct mem_cgroup **memcg,
 			bool oom)
 {
-	struct mem_cgroup *mem, *mem_over_limit;
+	struct mem_cgroup *mem, *mem_over_limit, *mem_over_soft_limit;
 	int nr_retries = MEM_CGROUP_RECLAIM_RETRIES;
-	struct res_counter *fail_res;
+	struct res_counter *fail_res, *soft_fail_res = NULL;
 
 	if (unlikely(test_thread_flag(TIF_MEMDIE))) {
 		/* Don't account this! */

mm/memcontrol.c static int __mem_cgroup_try_charge(struct mm_struct *mm, · line 1028

Quoted 6 lines

mm/memcontrol.c +2 -1

@@ -938,16 +1024,17 @@ static int __mem_cgroup_try_charge(struct mm_struct *mm,
 		int ret;
 		bool noswap = false;
 
-		ret = res_counter_charge(&mem->res, PAGE_SIZE, &fail_res);
+		ret = res_counter_charge(&mem->res, PAGE_SIZE, &fail_res,
+						&soft_fail_res);

As I pointed out, if this value is finally used once per HZ/X. checking this *alyways* is overkill. plz remove softlimit check from here.

Maybe code like this is good.
==

 if (need_softlimit_check(mem)) {
     softlimit_res = res_counter_check_under_softlimit(&mem->res);
     if (softlimit_res) {
        struct mem_cgroup *mem = mem_cgroup_from_cont(softlimit_res);
        update_tree()....      
     }

}
==

*And* what is important here is "need_softlimit_check(mem)".
As Andrew said, there may be something reasonable rather than using tick. So, adding "mem_cgroup_need_softlimit_check(mem)" and improving what it checks makes sense for development.

mm/memcontrol.c __mem_cgroup_uncharge_common(struct page *page, enum charge_type ctype) · line 1567

Quoted 12 lines

mm/memcontrol.c +2 -2

@@ -1461,9 +1560,9 @@ __mem_cgroup_uncharge_common(struct page *page, enum charge_type ctype)
 		break;
 	}
 
-	res_counter_uncharge(&mem->res, PAGE_SIZE);
+	res_counter_uncharge(&mem->res, PAGE_SIZE, &soft_limit_excess);
 	if (do_swap_account && (ctype != MEM_CGROUP_CHARGE_TYPE_SWAPOUT))
-		res_counter_uncharge(&mem->memsw, PAGE_SIZE);
+		res_counter_uncharge(&mem->memsw, PAGE_SIZE, NULL);
 	mem_cgroup_charge_statistics(mem, pc, false);
 

here, too.

Cound you add "mem_cgroup_need_softlimit_check(mem)" function here ? It will make code clearner, I think.

Quoted 10 lines · KAMEZAWA Hiroyuki

Maybe code like this is good.
==

 if (need_softlimit_check(mem)) {
     softlimit_res = res_counter_check_under_softlimit(&mem->res);
     if (softlimit_res) {
        struct mem_cgroup *mem = mem_cgroup_from_cont(softlimit_res);
        update_tree()....      
     }

}
==

An additional if is the problem?

We do all the checks under a lock we
already hold. I ran aim9, new_dbase, dbase, compute and shared tests to make sure that there is no degradation. I've not seen anything noticable so far.

Quoted 6 lines

*And* what is important here is "need_softlimit_check(mem)".
As Andrew said, there may be something reasonable rather than using tick. So, adding "mem_cgroup_need_softlimit_check(mem)" and improving what it checks makes sense for development.

OK, that is a good abstraction, but scanning as a metric does not guarantee anything. It is harder to come up with better heuristics with scan rate than to come up with something time based. I am open to suggestions for something reliable though.

 include/linux/memcontrol.h |    8 ++
 include/linux/swap.h       |    1 
 mm/memcontrol.c            |  205 ++++++++++++++++++++++++++++++++++++++++----
 mm/page_alloc.c            |    9 ++
 mm/vmscan.c                |    5 +
 5 files changed, 205 insertions(+), 23 deletions(-)

include/linux/memcontrol.h +7 -1

index 18146c9..b99d9c5 100644
@@ -116,7 +116,8 @@ static inline bool mem_cgroup_disabled(void)
 }
 
 extern bool mem_cgroup_oom_called(struct task_struct *task);
-
+unsigned long mem_cgroup_soft_limit_reclaim(struct zonelist *zl,
+						gfp_t gfp_mask);
 #else /* CONFIG_CGROUP_MEM_RES_CTLR */
 struct mem_cgroup;
 

@@ -264,6 +265,11 @@ mem_cgroup_print_oom_info(struct mem_cgroup *memcg, struct task_struct *p)
 {
 }
 
+static inline
+unsigned long mem_cgroup_soft_limit_reclaim(struct zonelist *zl, gfp_t gfp_mask)
+{
+	return 0;
+}
 #endif /* CONFIG_CGROUP_MEM_CONT */
 
 #endif /* _LINUX_MEMCONTROL_H */

include/linux/swap.h +1 -0

index 989eb53..c128337 100644
@@ -215,6 +215,7 @@ static inline void lru_cache_add_active_file(struct page *page)
 extern unsigned long try_to_free_pages(struct zonelist *zonelist, int order,
 					gfp_t gfp_mask);
 extern unsigned long try_to_free_mem_cgroup_pages(struct mem_cgroup *mem,
+						  struct zonelist *zl,
 						  gfp_t gfp_mask, bool noswap,
 						  unsigned int swappiness);
 extern int __isolate_lru_page(struct page *page, int mode, int file);

mm/memcontrol.c +1 -0

index 200d44a..980bd18 100644
@@ -191,6 +191,7 @@ struct mem_cgroup {
 	unsigned long last_tree_update;		/* Last time the tree was */
 						/* updated in jiffies     */
 
+	bool on_tree;				/* Is the node on tree? */
 	/*
 	 * statistics. This must be placed at the end of memcg.
 	 */

@@ -227,18 +228,29 @@ pcg_default_flags[NR_CHARGE_TYPE] = {
 #define MEMFILE_TYPE(val)	(((val) >> 16) & 0xffff)
 #define MEMFILE_ATTR(val)	((val) & 0xffff)
 

mm/memcontrol.c +59 -4

 static void mem_cgroup_get(struct mem_cgroup *mem);
 static void mem_cgroup_put(struct mem_cgroup *mem);
 static struct mem_cgroup *parent_mem_cgroup(struct mem_cgroup *mem);
 
-static void mem_cgroup_insert_exceeded(struct mem_cgroup *mem)
+static void __mem_cgroup_insert_exceeded(struct mem_cgroup *mem)
 {
 	struct rb_node **p = &mem_cgroup_soft_limit_tree.rb_node;
 	struct rb_node *parent = NULL;
 	struct mem_cgroup *mem_node;
-	unsigned long flags;
 
-	spin_lock_irqsave(&memcg_soft_limit_tree_lock, flags);
+	if (mem->on_tree)
+		return;
+
 	mem->usage_in_excess = res_counter_soft_limit_excess(&mem->res);
 	while (*p) {
 		parent = *p;

@@ -256,6 +268,23 @@ static void mem_cgroup_insert_exceeded(struct mem_cgroup *mem)
 	rb_insert_color(&mem->mem_cgroup_node,
 			&mem_cgroup_soft_limit_tree);
 	mem->last_tree_update = jiffies;
+	mem->on_tree = true;
+}
+
+static void __mem_cgroup_remove_exceeded(struct mem_cgroup *mem)
+{
+	if (!mem->on_tree)
+		return;
+	rb_erase(&mem->mem_cgroup_node, &mem_cgroup_soft_limit_tree);
+	mem->on_tree = false;
+}
+
+static void mem_cgroup_insert_exceeded(struct mem_cgroup *mem)
+{
+	unsigned long flags;
+
+	spin_lock_irqsave(&memcg_soft_limit_tree_lock, flags);
+	__mem_cgroup_insert_exceeded(mem);
 	spin_unlock_irqrestore(&memcg_soft_limit_tree_lock, flags);
 }
 

@@ -263,8 +292,53 @@ static void mem_cgroup_remove_exceeded(struct mem_cgroup *mem)
 {
 	unsigned long flags;
 	spin_lock_irqsave(&memcg_soft_limit_tree_lock, flags);
-	rb_erase(&mem->mem_cgroup_node, &mem_cgroup_soft_limit_tree);
+	__mem_cgroup_remove_exceeded(mem);
+	spin_unlock_irqrestore(&memcg_soft_limit_tree_lock, flags);
+}
+
+unsigned long mem_cgroup_get_excess(struct mem_cgroup *mem)
+{
+	unsigned long flags;
+	unsigned long long excess;
+
+	spin_lock_irqsave(&memcg_soft_limit_tree_lock, flags);
+	excess = mem->usage_in_excess >> PAGE_SHIFT;
 	spin_unlock_irqrestore(&memcg_soft_limit_tree_lock, flags);
+	return (excess > ULONG_MAX) ? ULONG_MAX : excess;
+}
+
+static struct mem_cgroup *__mem_cgroup_largest_soft_limit_node(void)
+{
+	struct rb_node *rightmost = NULL;
+	struct mem_cgroup *mem = NULL;
+
+retry:
+	rightmost = rb_last(&mem_cgroup_soft_limit_tree);
+	if (!rightmost)
+		goto done;		/* Nothing to reclaim from */
+
+	mem = rb_entry(rightmost, struct mem_cgroup, mem_cgroup_node);
+	/*
+	 * Remove the node now but someone else can add it back,
+	 * we will to add it back at the end of reclaim to its correct
+	 * position in the tree.
+	 */
+	__mem_cgroup_remove_exceeded(mem);
+	if (!css_tryget(&mem->css) || !res_counter_soft_limit_excess(&mem->res))
+		goto retry;
+done:
+	return mem;
+}
+
+static struct mem_cgroup *mem_cgroup_largest_soft_limit_node(void)

mm/memcontrol.c +10 -2

 static void mem_cgroup_charge_statistics(struct mem_cgroup *mem,

@@ -889,14 +963,42 @@ mem_cgroup_select_victim(struct mem_cgroup *root_mem)
  * If shrink==true, for avoiding to free too much, this returns immedieately.
  */
 static int mem_cgroup_hierarchical_reclaim(struct mem_cgroup *root_mem,
-				   gfp_t gfp_mask, bool noswap, bool shrink)
+						struct zonelist *zl,
+						gfp_t gfp_mask,
+						unsigned long reclaim_options)
 {
 	struct mem_cgroup *victim;
 	int ret, total = 0;
 	int loop = 0;
+	bool noswap = reclaim_options & MEM_CGROUP_RECLAIM_NOSWAP;
+	bool shrink = reclaim_options & MEM_CGROUP_RECLAIM_SHRINK;
+	bool check_soft = reclaim_options & MEM_CGROUP_RECLAIM_SOFT;
+	unsigned long excess = mem_cgroup_get_excess(root_mem);
 
-	while (loop < 2) {
+	while (1) {
+		if (loop >= 2) {
+			if (!check_soft)

mm/memcontrol.c +61 -12

 		victim = mem_cgroup_select_victim(root_mem);
+		/*
+		 * In the first loop, don't reclaim from victims below
+		 * their soft limit
+		 */
+		if (!loop && res_counter_check_under_soft_limit(&victim->res)) {
+			if (victim == root_mem)
+				loop++;
+			css_put(&victim->css);
+			continue;
+		}
 		if (victim == root_mem)
 			loop++;
 		if (!mem_cgroup_local_usage(&victim->stat)) {

@@ -905,8 +1007,9 @@ static int mem_cgroup_hierarchical_reclaim(struct mem_cgroup *root_mem,
 			continue;
 		}
 		/* we use swappiness of local cgroup */
-		ret = try_to_free_mem_cgroup_pages(victim, gfp_mask, noswap,
-						   get_swappiness(victim));
+		ret = try_to_free_mem_cgroup_pages(victim, zl, gfp_mask,
+							noswap,
+							get_swappiness(victim));
 		css_put(&victim->css);
 		/*
 		 * At shrinking usage, we can't check we should stop here or

@@ -916,7 +1019,10 @@ static int mem_cgroup_hierarchical_reclaim(struct mem_cgroup *root_mem,
 		if (shrink)
 			return ret;
 		total += ret;
-		if (mem_cgroup_check_under_limit(root_mem))
+		if (check_soft) {
+			if (res_counter_check_under_soft_limit(&root_mem->res))
+				return total;
+		} else if (mem_cgroup_check_under_limit(root_mem))
 			return 1 + total;
 	}
 	return total;

@@ -1022,7 +1128,7 @@ static int __mem_cgroup_try_charge(struct mm_struct *mm,
 
 	while (1) {
 		int ret;
-		bool noswap = false;
+		unsigned long flags = 0;
 
 		ret = res_counter_charge(&mem->res, PAGE_SIZE, &fail_res,
 						&soft_fail_res);

@@ -1035,7 +1141,7 @@ static int __mem_cgroup_try_charge(struct mm_struct *mm,
 				break;
 			/* mem+swap counter fails */
 			res_counter_uncharge(&mem->res, PAGE_SIZE, NULL);
-			noswap = true;
+			flags = MEM_CGROUP_RECLAIM_NOSWAP;
 			mem_over_limit = mem_cgroup_from_res_counter(fail_res,
 									memsw);
 		} else

@@ -1046,8 +1152,8 @@ static int __mem_cgroup_try_charge(struct mm_struct *mm,
 		if (!(gfp_mask & __GFP_WAIT))
 			goto nomem;
 
-		ret = mem_cgroup_hierarchical_reclaim(mem_over_limit, gfp_mask,
-							noswap, false);
+		ret = mem_cgroup_hierarchical_reclaim(mem_over_limit, NULL,
+							gfp_mask, flags);
 		if (ret)
 			continue;
 

@@ -1757,8 +1863,8 @@ int mem_cgroup_shrink_usage(struct page *page,
 		return 0;
 
 	do {
-		progress = mem_cgroup_hierarchical_reclaim(mem,
-					gfp_mask, true, false);
+		progress = mem_cgroup_hierarchical_reclaim(mem, NULL,
+					gfp_mask, MEM_CGROUP_RECLAIM_NOSWAP);
 		progress += mem_cgroup_check_under_limit(mem);
 	} while (!progress && --retry);
 

@@ -1812,8 +1918,9 @@ static int mem_cgroup_resize_limit(struct mem_cgroup *memcg,
 		if (!ret)
 			break;
 
-		progress = mem_cgroup_hierarchical_reclaim(memcg, GFP_KERNEL,
-						   false, true);
+		progress = mem_cgroup_hierarchical_reclaim(memcg, NULL,
+						GFP_KERNEL,
+						MEM_CGROUP_RECLAIM_SHRINK);
 		curusage = res_counter_read_u64(&memcg->res, RES_USAGE);
 		/* Usage is reduced ? */
   		if (curusage >= oldusage)

@@ -1861,7 +1968,9 @@ int mem_cgroup_resize_memsw_limit(struct mem_cgroup *memcg,
 		if (!ret)
 			break;
 
-		mem_cgroup_hierarchical_reclaim(memcg, GFP_KERNEL, true, true);
+		mem_cgroup_hierarchical_reclaim(memcg, NULL, GFP_KERNEL,
+						MEM_CGROUP_RECLAIM_NOSWAP |
+						MEM_CGROUP_RECLAIM_SHRINK);
 		curusage = res_counter_read_u64(&memcg->memsw, RES_USAGE);
 		/* Usage is reduced ? */
 		if (curusage >= oldusage)

@@ -1872,6 +1981,62 @@ int mem_cgroup_resize_memsw_limit(struct mem_cgroup *memcg,
 	return ret;
 }
 
+unsigned long mem_cgroup_soft_limit_reclaim(struct zonelist *zl, gfp_t gfp_mask)
+{
+	unsigned long nr_reclaimed = 0;
+	struct mem_cgroup *mem, *next_mem = NULL;
+	unsigned long flags;
+	unsigned long reclaimed;
+
+	/*
+	 * This loop can run a while, specially if mem_cgroup's continuously
+	 * keep exceeding their soft limit and putting the system under
+	 * pressure
+	 */
+	do {
+		if (next_mem)
+			mem = next_mem;
+		else
+			mem = mem_cgroup_largest_soft_limit_node();
+		if (!mem)
+			break;
+
+		reclaimed = mem_cgroup_hierarchical_reclaim(mem, zl,
+						gfp_mask,
+						MEM_CGROUP_RECLAIM_SOFT);
+		nr_reclaimed += reclaimed;
+		spin_lock_irqsave(&memcg_soft_limit_tree_lock, flags);
+
+		/*
+		 * If we failed to reclaim anything from this memory cgroup
+		 * it is time to move on to the next cgroup
+		 */
+		next_mem = NULL;
+		if (!reclaimed) {

mm/memcontrol.c +9 -1

+		spin_unlock_irqrestore(&memcg_soft_limit_tree_lock, flags);
+		css_put(&mem->css);
+	} while (!nr_reclaimed);
+	return nr_reclaimed;
+}
+
 /*
  * This routine traverse page_cgroup in given list and drop them all.
  * *And* this routine doesn't reclaim page itself, just removes page_cgroup.

@@ -1995,7 +2160,7 @@ try_to_free:
 			ret = -EINTR;
 			goto out;
 		}
-		progress = try_to_free_mem_cgroup_pages(mem, GFP_KERNEL,
+		progress = try_to_free_mem_cgroup_pages(mem, NULL, GFP_KERNEL,
 						false, get_swappiness(mem));
 		if (!progress) {
 			nr_retries--;

@@ -2600,6 +2765,8 @@ mem_cgroup_create(struct cgroup_subsys *ss, struct cgroup *cont)
 	mem->last_scanned_child = 0;
 	mem->usage_in_excess = 0;
 	mem->last_tree_update = 0;	/* Yes, time begins at 0 here */
+	mem->on_tree = false;
+
 	spin_lock_init(&mem->reclaim_param_lock);
 
 	if (parent)

mm/page_alloc.c +0 -0

index f8fd1e2..5e1a6ca 100644
@@ -1598,7 +1598,14 @@ nofail_alloc:
 	reclaim_state.reclaimed_slab = 0;
 	p->reclaim_state = &reclaim_state;

Quoted 63 lines · On Sat, 14 Mar 2009 23:01:11 +0530 Balbir Singh <[email protected]>…

 include/linux/memcontrol.h |    8 ++
 include/linux/swap.h       |    1 
 mm/memcontrol.c            |  205 ++++++++++++++++++++++++++++++++++++++++----
 mm/page_alloc.c            |    9 ++
 mm/vmscan.c                |    5 +
 5 files changed, 205 insertions(+), 23 deletions(-)

include/linux/memcontrol.h +7 -1

index 18146c9..b99d9c5 100644
@@ -116,7 +116,8 @@ static inline bool mem_cgroup_disabled(void)
 }
 
 extern bool mem_cgroup_oom_called(struct task_struct *task);
-
+unsigned long mem_cgroup_soft_limit_reclaim(struct zonelist *zl,
+						gfp_t gfp_mask);
 #else /* CONFIG_CGROUP_MEM_RES_CTLR */
 struct mem_cgroup;
 

@@ -264,6 +265,11 @@ mem_cgroup_print_oom_info(struct mem_cgroup *memcg, struct task_struct *p)
 {
 }
 
+static inline
+unsigned long mem_cgroup_soft_limit_reclaim(struct zonelist *zl, gfp_t gfp_mask)
+{
+	return 0;
+}
 #endif /* CONFIG_CGROUP_MEM_CONT */
 
 #endif /* _LINUX_MEMCONTROL_H */

include/linux/swap.h +1 -0

index 989eb53..c128337 100644
@@ -215,6 +215,7 @@ static inline void lru_cache_add_active_file(struct page *page)
 extern unsigned long try_to_free_pages(struct zonelist *zonelist, int order,
 					gfp_t gfp_mask);
 extern unsigned long try_to_free_mem_cgroup_pages(struct mem_cgroup *mem,
+						  struct zonelist *zl,
 						  gfp_t gfp_mask, bool noswap,
 						  unsigned int swappiness);
 extern int __isolate_lru_page(struct page *page, int mode, int file);

mm/memcontrol.c +1 -0

index 200d44a..980bd18 100644
@@ -191,6 +191,7 @@ struct mem_cgroup {
 	unsigned long last_tree_update;		/* Last time the tree was */
 						/* updated in jiffies     */
 
+	bool on_tree;				/* Is the node on tree? */
 	/*
 	 * statistics. This must be placed at the end of memcg.
 	 */

@@ -227,18 +228,29 @@ pcg_default_flags[NR_CHARGE_TYPE] = {
 #define MEMFILE_TYPE(val)	(((val) >> 16) & 0xffff)
 #define MEMFILE_ATTR(val)	((val) & 0xffff)
 

mm/memcontrol.c pcg_default_flags[NR_CHARGE_TYPE] = { · line 240

Quoted 10 lines

mm/memcontrol.c +10 -0

+/*
+ * Bits used for hierarchical reclaim bits
+ */
+#define MEM_CGROUP_RECLAIM_NOSWAP_BIT	0x0
+#define MEM_CGROUP_RECLAIM_NOSWAP	(1 << MEM_CGROUP_RECLAIM_NOSWAP_BIT)
+#define MEM_CGROUP_RECLAIM_SHRINK_BIT	0x1
+#define MEM_CGROUP_RECLAIM_SHRINK	(1 << MEM_CGROUP_RECLAIM_SHRINK_BIT)
+#define MEM_CGROUP_RECLAIM_SOFT_BIT	0x2
+#define MEM_CGROUP_RECLAIM_SOFT		(1 << MEM_CGROUP_RECLAIM_SOFT_BIT)
+

Could you divide this clean-up part to other patch ?

Quoted 92 lines

mm/memcontrol.c +63 -4

 static void mem_cgroup_get(struct mem_cgroup *mem);
 static void mem_cgroup_put(struct mem_cgroup *mem);
 static struct mem_cgroup *parent_mem_cgroup(struct mem_cgroup *mem);
 
-static void mem_cgroup_insert_exceeded(struct mem_cgroup *mem)
+static void __mem_cgroup_insert_exceeded(struct mem_cgroup *mem)
 {
 	struct rb_node **p = &mem_cgroup_soft_limit_tree.rb_node;
 	struct rb_node *parent = NULL;
 	struct mem_cgroup *mem_node;
-	unsigned long flags;
 
-	spin_lock_irqsave(&memcg_soft_limit_tree_lock, flags);
+	if (mem->on_tree)
+		return;
+
 	mem->usage_in_excess = res_counter_soft_limit_excess(&mem->res);
 	while (*p) {
 		parent = *p;

@@ -256,6 +268,23 @@ static void mem_cgroup_insert_exceeded(struct mem_cgroup *mem)
 	rb_insert_color(&mem->mem_cgroup_node,
 			&mem_cgroup_soft_limit_tree);
 	mem->last_tree_update = jiffies;
+	mem->on_tree = true;
+}
+
+static void __mem_cgroup_remove_exceeded(struct mem_cgroup *mem)
+{
+	if (!mem->on_tree)
+		return;
+	rb_erase(&mem->mem_cgroup_node, &mem_cgroup_soft_limit_tree);
+	mem->on_tree = false;
+}
+
+static void mem_cgroup_insert_exceeded(struct mem_cgroup *mem)
+{
+	unsigned long flags;
+
+	spin_lock_irqsave(&memcg_soft_limit_tree_lock, flags);
+	__mem_cgroup_insert_exceeded(mem);
 	spin_unlock_irqrestore(&memcg_soft_limit_tree_lock, flags);
 }
 

@@ -263,8 +292,53 @@ static void mem_cgroup_remove_exceeded(struct mem_cgroup *mem)
 {
 	unsigned long flags;
 	spin_lock_irqsave(&memcg_soft_limit_tree_lock, flags);
-	rb_erase(&mem->mem_cgroup_node, &mem_cgroup_soft_limit_tree);
+	__mem_cgroup_remove_exceeded(mem);
+	spin_unlock_irqrestore(&memcg_soft_limit_tree_lock, flags);
+}
+
+unsigned long mem_cgroup_get_excess(struct mem_cgroup *mem)
+{
+	unsigned long flags;
+	unsigned long long excess;
+
+	spin_lock_irqsave(&memcg_soft_limit_tree_lock, flags);
+	excess = mem->usage_in_excess >> PAGE_SHIFT;
 	spin_unlock_irqrestore(&memcg_soft_limit_tree_lock, flags);
+	return (excess > ULONG_MAX) ? ULONG_MAX : excess;
+}
+
+static struct mem_cgroup *__mem_cgroup_largest_soft_limit_node(void)
+{
+	struct rb_node *rightmost = NULL;
+	struct mem_cgroup *mem = NULL;
+
+retry:
+	rightmost = rb_last(&mem_cgroup_soft_limit_tree);
+	if (!rightmost)
+		goto done;		/* Nothing to reclaim from */
+
+	mem = rb_entry(rightmost, struct mem_cgroup, mem_cgroup_node);
+	/*
+	 * Remove the node now but someone else can add it back,
+	 * we will to add it back at the end of reclaim to its correct
+	 * position in the tree.
+	 */
+	__mem_cgroup_remove_exceeded(mem);
+	if (!css_tryget(&mem->css) || !res_counter_soft_limit_excess(&mem->res))
+		goto retry;
+done:
+	return mem;
+}
+
+static struct mem_cgroup *mem_cgroup_largest_soft_limit_node(void)
+{
+	struct mem_cgroup *mem;
+	unsigned long flags;
+

mm/memcontrol.c static void mem_cgroup_remove_exceeded(struct mem_cgroup *mem) · line 343

Quoted 6 lines

mm/memcontrol.c +4 -0

+	spin_lock_irqsave(&memcg_soft_limit_tree_lock, flags);
+	mem = __mem_cgroup_largest_soft_limit_node();
+	spin_unlock_irqrestore(&memcg_soft_limit_tree_lock, flags);
+	return mem;
 }
 

Can you think of avoiding this global-lock ?(As Kosaki said.) IIUC, cpu-scheduler's RB tree/hrtimer's one, you memtioned, is per-cpu.

Quoted 28 lines

mm/memcontrol.c +15 -2

 static void mem_cgroup_charge_statistics(struct mem_cgroup *mem,

@@ -889,14 +963,42 @@ mem_cgroup_select_victim(struct mem_cgroup *root_mem)
  * If shrink==true, for avoiding to free too much, this returns immedieately.
  */
 static int mem_cgroup_hierarchical_reclaim(struct mem_cgroup *root_mem,
-				   gfp_t gfp_mask, bool noswap, bool shrink)
+						struct zonelist *zl,
+						gfp_t gfp_mask,
+						unsigned long reclaim_options)
 {
 	struct mem_cgroup *victim;
 	int ret, total = 0;
 	int loop = 0;
+	bool noswap = reclaim_options & MEM_CGROUP_RECLAIM_NOSWAP;
+	bool shrink = reclaim_options & MEM_CGROUP_RECLAIM_SHRINK;
+	bool check_soft = reclaim_options & MEM_CGROUP_RECLAIM_SOFT;
+	unsigned long excess = mem_cgroup_get_excess(root_mem);
 
-	while (loop < 2) {
+	while (1) {
+		if (loop >= 2) {
+			if (!check_soft)
+				break;
+			/*
+			 * We want to do more targetted reclaim. excess >> 4
+			 * >> 4 is not to excessive so as to reclaim too
+			 * much, nor too less that we keep coming back

mm/memcontrol.c mem_cgroup_select_victim(struct mem_cgroup *root_mem) · line 990

Quoted 5 lines

mm/memcontrol.c +5 -0

+			 * to reclaim from this cgroup
+			 */
+			if (total >= (excess >> 4))
+				break;
+		}

I wonder this means, in very bad case, the thread cannot exit this loop... right ?

Quoted 140 lines

mm/memcontrol.c +64 -12

 		victim = mem_cgroup_select_victim(root_mem);
+		/*
+		 * In the first loop, don't reclaim from victims below
+		 * their soft limit
+		 */
+		if (!loop && res_counter_check_under_soft_limit(&victim->res)) {
+			if (victim == root_mem)
+				loop++;
+			css_put(&victim->css);
+			continue;
+		}
 		if (victim == root_mem)
 			loop++;
 		if (!mem_cgroup_local_usage(&victim->stat)) {

@@ -905,8 +1007,9 @@ static int mem_cgroup_hierarchical_reclaim(struct mem_cgroup *root_mem,
 			continue;
 		}
 		/* we use swappiness of local cgroup */
-		ret = try_to_free_mem_cgroup_pages(victim, gfp_mask, noswap,
-						   get_swappiness(victim));
+		ret = try_to_free_mem_cgroup_pages(victim, zl, gfp_mask,
+							noswap,
+							get_swappiness(victim));
 		css_put(&victim->css);
 		/*
 		 * At shrinking usage, we can't check we should stop here or

@@ -916,7 +1019,10 @@ static int mem_cgroup_hierarchical_reclaim(struct mem_cgroup *root_mem,
 		if (shrink)
 			return ret;
 		total += ret;
-		if (mem_cgroup_check_under_limit(root_mem))
+		if (check_soft) {
+			if (res_counter_check_under_soft_limit(&root_mem->res))
+				return total;
+		} else if (mem_cgroup_check_under_limit(root_mem))
 			return 1 + total;
 	}
 	return total;

@@ -1022,7 +1128,7 @@ static int __mem_cgroup_try_charge(struct mm_struct *mm,
 
 	while (1) {
 		int ret;
-		bool noswap = false;
+		unsigned long flags = 0;
 
 		ret = res_counter_charge(&mem->res, PAGE_SIZE, &fail_res,
 						&soft_fail_res);

@@ -1035,7 +1141,7 @@ static int __mem_cgroup_try_charge(struct mm_struct *mm,
 				break;
 			/* mem+swap counter fails */
 			res_counter_uncharge(&mem->res, PAGE_SIZE, NULL);
-			noswap = true;
+			flags = MEM_CGROUP_RECLAIM_NOSWAP;
 			mem_over_limit = mem_cgroup_from_res_counter(fail_res,
 									memsw);
 		} else

@@ -1046,8 +1152,8 @@ static int __mem_cgroup_try_charge(struct mm_struct *mm,
 		if (!(gfp_mask & __GFP_WAIT))
 			goto nomem;
 
-		ret = mem_cgroup_hierarchical_reclaim(mem_over_limit, gfp_mask,
-							noswap, false);
+		ret = mem_cgroup_hierarchical_reclaim(mem_over_limit, NULL,
+							gfp_mask, flags);
 		if (ret)
 			continue;
 

@@ -1757,8 +1863,8 @@ int mem_cgroup_shrink_usage(struct page *page,
 		return 0;
 
 	do {
-		progress = mem_cgroup_hierarchical_reclaim(mem,
-					gfp_mask, true, false);
+		progress = mem_cgroup_hierarchical_reclaim(mem, NULL,
+					gfp_mask, MEM_CGROUP_RECLAIM_NOSWAP);
 		progress += mem_cgroup_check_under_limit(mem);
 	} while (!progress && --retry);
 

@@ -1812,8 +1918,9 @@ static int mem_cgroup_resize_limit(struct mem_cgroup *memcg,
 		if (!ret)
 			break;
 
-		progress = mem_cgroup_hierarchical_reclaim(memcg, GFP_KERNEL,
-						   false, true);
+		progress = mem_cgroup_hierarchical_reclaim(memcg, NULL,
+						GFP_KERNEL,
+						MEM_CGROUP_RECLAIM_SHRINK);
 		curusage = res_counter_read_u64(&memcg->res, RES_USAGE);
 		/* Usage is reduced ? */
   		if (curusage >= oldusage)

@@ -1861,7 +1968,9 @@ int mem_cgroup_resize_memsw_limit(struct mem_cgroup *memcg,
 		if (!ret)
 			break;
 
-		mem_cgroup_hierarchical_reclaim(memcg, GFP_KERNEL, true, true);
+		mem_cgroup_hierarchical_reclaim(memcg, NULL, GFP_KERNEL,
+						MEM_CGROUP_RECLAIM_NOSWAP |
+						MEM_CGROUP_RECLAIM_SHRINK);
 		curusage = res_counter_read_u64(&memcg->memsw, RES_USAGE);
 		/* Usage is reduced ? */
 		if (curusage >= oldusage)

@@ -1872,6 +1981,62 @@ int mem_cgroup_resize_memsw_limit(struct mem_cgroup *memcg,
 	return ret;
 }
 
+unsigned long mem_cgroup_soft_limit_reclaim(struct zonelist *zl, gfp_t gfp_mask)
+{
+	unsigned long nr_reclaimed = 0;
+	struct mem_cgroup *mem, *next_mem = NULL;
+	unsigned long flags;
+	unsigned long reclaimed;
+
+	/*
+	 * This loop can run a while, specially if mem_cgroup's continuously
+	 * keep exceeding their soft limit and putting the system under
+	 * pressure
+	 */
+	do {
+		if (next_mem)
+			mem = next_mem;
+		else
+			mem = mem_cgroup_largest_soft_limit_node();
+		if (!mem)
+			break;
+
+		reclaimed = mem_cgroup_hierarchical_reclaim(mem, zl,
+						gfp_mask,
+						MEM_CGROUP_RECLAIM_SOFT);
+		nr_reclaimed += reclaimed;
+		spin_lock_irqsave(&memcg_soft_limit_tree_lock, flags);
+
+		/*
+		 * If we failed to reclaim anything from this memory cgroup
+		 * it is time to move on to the next cgroup
+		 */
+		next_mem = NULL;
+		if (!reclaimed) {
+			do {
+				/*
+				 * By the time we get the soft_limit lock

mm/memcontrol.c int mem_cgroup_resize_memsw_limit(struct mem_cgroup *memcg, · line 2025

Quoted 7 lines

mm/memcontrol.c +7 -0

+				 * again, someone might have aded the
+				 * group back on the RB tree. Iterate to
+				 * make sure we get a different mem.
+				 * mem_cgroup_largest_soft_limit_node returns
+				 * NULL if no other cgroup is present on
+				 * the tree
+				 */

Do we have to allow "someone will push back" case ?

Quoted 4 lines

mm/memcontrol.c +4 -0

+				next_mem =
+					__mem_cgroup_largest_soft_limit_node();
+			} while (next_mem == mem);
+		}

mm/memcontrol.c int mem_cgroup_resize_memsw_limit(struct mem_cgroup *memcg, · line 2033

Quoted 4 lines

mm/memcontrol.c +4 -0

+		mem->usage_in_excess = res_counter_soft_limit_excess(&mem->res);
+		__mem_cgroup_remove_exceeded(mem);
+		if (mem->usage_in_excess)
+			__mem_cgroup_insert_exceeded(mem);

If next_mem == NULL here, (means "mem" is an only mem_cgroup which excess softlimit.) mem will be found again even if !reclaimed.
plz check.

Quoted 36 lines

mm/memcontrol.c +9 -1

+		spin_unlock_irqrestore(&memcg_soft_limit_tree_lock, flags);
+		css_put(&mem->css);
+	} while (!nr_reclaimed);
+	return nr_reclaimed;
+}
+
 /*
  * This routine traverse page_cgroup in given list and drop them all.
  * *And* this routine doesn't reclaim page itself, just removes page_cgroup.

@@ -1995,7 +2160,7 @@ try_to_free:
 			ret = -EINTR;
 			goto out;
 		}
-		progress = try_to_free_mem_cgroup_pages(mem, GFP_KERNEL,
+		progress = try_to_free_mem_cgroup_pages(mem, NULL, GFP_KERNEL,
 						false, get_swappiness(mem));
 		if (!progress) {
 			nr_retries--;

@@ -2600,6 +2765,8 @@ mem_cgroup_create(struct cgroup_subsys *ss, struct cgroup *cont)
 	mem->last_scanned_child = 0;
 	mem->usage_in_excess = 0;
 	mem->last_tree_update = 0;	/* Yes, time begins at 0 here */
+	mem->on_tree = false;
+
 	spin_lock_init(&mem->reclaim_param_lock);
 
 	if (parent)

mm/page_alloc.c +0 -0

index f8fd1e2..5e1a6ca 100644
@@ -1598,7 +1598,14 @@ nofail_alloc:
 	reclaim_state.reclaimed_slab = 0;
 	p->reclaim_state = &reclaim_state;
 

mm/page_alloc.c nofail_alloc: · line 1608

Quoted 9 lines

mm/page_alloc.c +8 -1

-	did_some_progress = try_to_free_pages(zonelist, order, gfp_mask);
+	/*
+	 * Try to free up some pages from the memory controllers soft
+	 * limit queue.
+	 */
+	did_some_progress = mem_cgroup_soft_limit_reclaim(zonelist, gfp_mask);
+	if (order || !did_some_progress)
+		did_some_progress += try_to_free_pages(zonelist, order,
+							gfp_mask);

I'm not sure but do we have to call try_to_free()...twice ?

if (order)
did_some_progress = mem_cgroup_soft_limit_reclaim(zonelist, gfp_mask);
if (!order || did_some_progrees)
did_some_progress = mem_cgroup_soft_limit_reclaim(zonelist, gfp_mask);

IIRC, why Kosaki said "don't check order" is because this was called by kswapd() case.

BTW, mem_cgroup_soft_limit_reclaim() can do enough job even under (gfp_mask & (__GFP_IO|__GFP_FS)) == 0 case ?

Quoted 28 lines

Quoted 28 lines

static void mem_cgroup_charge_statistics(struct mem_cgroup *mem,

mm/memcontrol.c +15 -2

@@ -889,14 +963,42 @@ mem_cgroup_select_victim(struct mem_cgroup *root_mem)
  * If shrink==true, for avoiding to free too much, this returns immedieately.
  */
 static int mem_cgroup_hierarchical_reclaim(struct mem_cgroup *root_mem,
-				   gfp_t gfp_mask, bool noswap, bool shrink)
+						struct zonelist *zl,
+						gfp_t gfp_mask,
+						unsigned long reclaim_options)
 {
 	struct mem_cgroup *victim;
 	int ret, total = 0;
 	int loop = 0;
+	bool noswap = reclaim_options & MEM_CGROUP_RECLAIM_NOSWAP;
+	bool shrink = reclaim_options & MEM_CGROUP_RECLAIM_SHRINK;
+	bool check_soft = reclaim_options & MEM_CGROUP_RECLAIM_SOFT;
+	unsigned long excess = mem_cgroup_get_excess(root_mem);
 
-	while (loop < 2) {
+	while (1) {
+		if (loop >= 2) {
+			if (!check_soft)
+				break;
+			/*
+			 * We want to do more targetted reclaim. excess >> 4
+			 * >> 4 is not to excessive so as to reclaim too
+			 * much, nor too less that we keep coming back

Quoted 8 lines

mm/memcontrol.c mem_cgroup_select_victim(struct mem_cgroup *root_mem) · line 990

Quoted 5 lines

mm/memcontrol.c +5 -0

+			 * to reclaim from this cgroup
+			 */
+			if (total >= (excess >> 4))
+				break;
+		}

I wonder this means, in very bad case, the thread cannot exit this loop... right ?

Potentially. When we do force empty, we actually reclaim all pages in a loop. Do you want to see additional checks here?

Quoted 4 lines

Quoted 4 lines

mm/memcontrol.c +4 -0

+		if (!reclaimed) {
+			do {
+				/*
+				 * By the time we get the soft_limit lock

Quoted 9 lines

mm/memcontrol.c int mem_cgroup_resize_memsw_limit(struct mem_cgroup *memcg, · line 2025

Quoted 7 lines

mm/memcontrol.c +7 -0

+				 * again, someone might have aded the
+				 * group back on the RB tree. Iterate to
+				 * make sure we get a different mem.
+				 * mem_cgroup_largest_soft_limit_node returns
+				 * NULL if no other cgroup is present on
+				 * the tree
+				 */

Do we have to allow "someone will push back" case ?

Not sure I understand your comment completely? When you say push back, are you referring to some one else adding back the RB-Tree to the node?

If so, yes, that is quite possible and I've seen it happen.

Quoted 4 lines

Quoted 4 lines

mm/memcontrol.c +4 -0

+				next_mem =
+					__mem_cgroup_largest_soft_limit_node();
+			} while (next_mem == mem);
+		}

Quoted 8 lines

mm/memcontrol.c int mem_cgroup_resize_memsw_limit(struct mem_cgroup *memcg, · line 2033

Quoted 4 lines

mm/memcontrol.c +4 -0

+		mem->usage_in_excess = res_counter_soft_limit_excess(&mem->res);
+		__mem_cgroup_remove_exceeded(mem);
+		if (mem->usage_in_excess)
+			__mem_cgroup_insert_exceeded(mem);

If next_mem == NULL here, (means "mem" is an only mem_cgroup which excess softlimit.) mem will be found again even if !reclaimed.
plz check.

Yes, We need to add a if (!next_mem) break; Thanks!

Quoted 36 lines

Quoted 36 lines

mm/memcontrol.c +9 -1

+		spin_unlock_irqrestore(&memcg_soft_limit_tree_lock, flags);
+		css_put(&mem->css);
+	} while (!nr_reclaimed);
+	return nr_reclaimed;
+}
+
 /*
  * This routine traverse page_cgroup in given list and drop them all.
  * *And* this routine doesn't reclaim page itself, just removes page_cgroup.

@@ -1995,7 +2160,7 @@ try_to_free:
 			ret = -EINTR;
 			goto out;
 		}
-		progress = try_to_free_mem_cgroup_pages(mem, GFP_KERNEL,
+		progress = try_to_free_mem_cgroup_pages(mem, NULL, GFP_KERNEL,
 						false, get_swappiness(mem));
 		if (!progress) {
 			nr_retries--;

@@ -2600,6 +2765,8 @@ mem_cgroup_create(struct cgroup_subsys *ss, struct cgroup *cont)
 	mem->last_scanned_child = 0;
 	mem->usage_in_excess = 0;
 	mem->last_tree_update = 0;	/* Yes, time begins at 0 here */
+	mem->on_tree = false;
+
 	spin_lock_init(&mem->reclaim_param_lock);
 
 	if (parent)

mm/page_alloc.c +0 -0

index f8fd1e2..5e1a6ca 100644
@@ -1598,7 +1598,14 @@ nofail_alloc:
 	reclaim_state.reclaimed_slab = 0;
 	p->reclaim_state = &reclaim_state;
 

Quoted 10 lines

mm/page_alloc.c nofail_alloc: · line 1608

Quoted 9 lines

mm/page_alloc.c +8 -1

-	did_some_progress = try_to_free_pages(zonelist, order, gfp_mask);
+	/*
+	 * Try to free up some pages from the memory controllers soft
+	 * limit queue.
+	 */
+	did_some_progress = mem_cgroup_soft_limit_reclaim(zonelist, gfp_mask);
+	if (order || !did_some_progress)
+		did_some_progress += try_to_free_pages(zonelist, order,
+							gfp_mask);

I'm not sure but do we have to call try_to_free()...twice ?

We call it twice, once for the memory controller and once for normal reclaim (try_to_free_mem_cgroup_pages() and try_to_free_pages()), is that an issue?

Quoted 6 lines

if (order)
did_some_progress = mem_cgroup_soft_limit_reclaim(zonelist, gfp_mask);
if (!order || did_some_progrees)
did_some_progress = mem_cgroup_soft_limit_reclaim(zonelist, gfp_mask);

I don't understand the code snippet above.

Quoted 5 lines

IIRC, why Kosaki said "don't check order" is because this was called by kswapd() case.

BTW, mem_cgroup_soft_limit_reclaim() can do enough job even under (gfp_mask & (__GFP_IO|__GFP_FS)) == 0 case ?

What about clean page cache? Anyway, we pass the gfp_mask, so the reclaimer knows what pages to reclaim from, so it should return quickly if it can't reclaim. Am I missing something?

On Mon, 16 Mar 2009 14:05:12 +0530
Balbir Singh <[email protected]> wrote:

For example, shrink_slab() is not called. and this must be called.

For exmaple, we may have to add
sc->call_shrink_slab
flag and set it "true" at soft limit reclaim.

Quoted 2 lines · On Mon, 16 Mar 2009 17:49:43 +0900 KAMEZAWA Hiroyuki…

On Mon, 16 Mar 2009 14:05:12 +0530
Balbir Singh <[email protected]> wrote:

Quoted 6 lines

For example, shrink_slab() is not called. and this must be called.

For exmaple, we may have to add
sc->call_shrink_slab
flag and set it "true" at soft limit reclaim.

At least, this check will be necessary in v7, I think. shrink_slab() should be called.

* KAMEZAWA Hiroyuki <[email protected]> [2009-03-16 18:03:08]:

Quoted 12 lines

Quoted 9 lines · On Mon, 16 Mar 2009 17:49:43 +0900 KAMEZAWA Hiroyuki…

On Mon, 16 Mar 2009 14:05:12 +0530
Balbir Singh <[email protected]> wrote:

For example, shrink_slab() is not called. and this must be called.

For exmaple, we may have to add
sc->call_shrink_slab
flag and set it "true" at soft limit reclaim.

Quoted 2 lines

At least, this check will be necessary in v7, I think. shrink_slab() should be called.

Why do you think so? So here is the design

1. If a cgroup was using over its soft limit, we believe that this cgroup created overall memory contention and caused the page reclaimer to get activated.

If we can solve the situation by
reclaiming from this cgroup, why do we need to invoke shrink_slab?

If the concern is that we are not following the traditional reclaim, soft limit reclaim can be followed by unconditional reclaim, but I believe this is not necessary. Remember, we wake up kswapd that will call shrink_slab if needed.

Quoted 15 lines · Balbir Singh

* KAMEZAWA Hiroyuki <[email protected]> [2009-03-16 18:03:08]:

Quoted 12 lines

Quoted 9 lines · On Mon, 16 Mar 2009 17:49:43 +0900 KAMEZAWA Hiroyuki…

On Mon, 16 Mar 2009 14:05:12 +0530
Balbir Singh <[email protected]> wrote:

For example, shrink_slab() is not called. and this must be called.

For exmaple, we may have to add
sc->call_shrink_slab
flag and set it "true" at soft limit reclaim.

Quoted 8 lines

Quoted 2 lines

At least, this check will be necessary in v7, I think. shrink_slab() should be called.

Why do you think so? So here is the design

1. If a cgroup was using over its soft limit, we believe that this cgroup created overall memory contention and caused the page reclaimer to get activated.

This assumption is wrong, see below.

Quoted 3 lines

If we can solve the situation by
reclaiming from this cgroup, why do we need to invoke shrink_slab?

No,
IIUC, in big server, inode, dentry cache etc....can occupy Gigabytes of memory even if 99% of them are not used.

By shrink_slab(), we can reclaim unused but cached slabs and make the kernel more healthy.

Quoted 5 lines

If the concern is that we are not following the traditional reclaim, soft limit reclaim can be followed by unconditional reclaim, but I believe this is not necessary. Remember, we wake up kswapd that will call shrink_slab if needed.

kswapd doesn't call shrink_slab() when zone->free is enough. (when direct recail did good jobs.)

Anyway, we'll have to add softlimit hook to kswapd.
I think you read Kosaki's e-mail to you.
==
in global reclaim view, foreground reclaim and background reclaim's
reclaim rate is about 1:9 typically.
==

* KAMEZAWA Hiroyuki <[email protected]> [2009-03-16 20:10:41]:

Quoted 17 lines

Quoted 8 lines

Quoted 2 lines

At least, this check will be necessary in v7, I think. shrink_slab() should be called.

Why do you think so? So here is the design

1. If a cgroup was using over its soft limit, we believe that this cgroup created overall memory contention and caused the page reclaimer to get activated.

This assumption is wrong, see below.

Quoted 3 lines

If we can solve the situation by
reclaiming from this cgroup, why do we need to invoke shrink_slab?

No,
IIUC, in big server, inode, dentry cache etc....can occupy Gigabytes of memory even if 99% of them are not used.

Quoted 3 lines

By shrink_slab(), we can reclaim unused but cached slabs and make the kernel more healthy.

But that is not the job of the soft limit reclaimer.. Yes if no groups are over their soft limit, the regular action will take place.

Quoted 5 lines

Quoted 5 lines

If the concern is that we are not following the traditional reclaim, soft limit reclaim can be followed by unconditional reclaim, but I believe this is not necessary. Remember, we wake up kswapd that will call shrink_slab if needed.

Quoted 3 lines

kswapd doesn't call shrink_slab() when zone->free is enough. (when direct recail did good jobs.)

If zone->free is high why do we need shrink_slab()? The other way of asking it is, why does the soft limit reclaimer need to call shrink_slab(), when its job is to reclaim memory from cgroups above their soft limits.

Quoted 19 lines · Balbir Singh

* KAMEZAWA Hiroyuki <[email protected]> [2009-03-16 20:10:41]:

Quoted 17 lines

Quoted 8 lines

Quoted 2 lines

At least, this check will be necessary in v7, I think. shrink_slab() should be called.

Why do you think so? So here is the design

1. If a cgroup was using over its soft limit, we believe that this cgroup created overall memory contention and caused the page reclaimer to get activated.

This assumption is wrong, see below.

Quoted 3 lines

If we can solve the situation by
reclaiming from this cgroup, why do we need to invoke shrink_slab?

No,
IIUC, in big server, inode, dentry cache etc....can occupy Gigabytes of memory even if 99% of them are not used.

Quoted 7 lines

Quoted 3 lines

By shrink_slab(), we can reclaim unused but cached slabs and make the kernel more healthy.

But that is not the job of the soft limit reclaimer.. Yes if no groups are over their soft limit, the regular action will take place.

Oh, yes, it's not job of memcg but it's job of memory management.

Quoted 9 lines

Quoted 8 lines

Quoted 5 lines

If the concern is that we are not following the traditional reclaim, soft limit reclaim can be followed by unconditional reclaim, but I believe this is not necessary. Remember, we wake up kswapd that will call shrink_slab if needed.

kswapd doesn't call shrink_slab() when zone->free is enough. (when direct recail did good jobs.)

Quoted 5 lines

If zone->free is high why do we need shrink_slab()? The other way of asking it is, why does the soft limit reclaimer need to call shrink_slab(), when its job is to reclaim memory from cgroups above their soft limits.

Why do you consider that softlimit is called more than necessary if shrink_slab() is never called ?

* KAMEZAWA Hiroyuki <[email protected]> [2009-03-16 20:58:30]:

A run away application can do that. Like I mentioned with the tests I did for your patches. Soft limits were at 1G/2G and the applications (two) tried to touch all the memory in the system. The point is that shrink_slab() will be called if the normal system experiences watermark issues, soft limits will tackle/control cgroups running out of their soft limits and causing memory contention to take place.

Quoted 9 lines · On Mon, 16 Mar 2009 17:49:15 +0530 Balbir Singh <[email protected]>…

* KAMEZAWA Hiroyuki <[email protected]> [2009-03-16 20:58:30]:

A run away application can do that. Like I mentioned with the tests I did for your patches. Soft limits were at 1G/2G and the applications (two) tried to touch all the memory in the system. The point is that shrink_slab() will be called if the normal system experiences watermark issues, soft limits will tackle/control cgroups running out of their soft limits and causing memory contention to take place.

Ok, then, how about this ?

Because our target is "softlimit" and not "memory shortage"
- don't call soft limit from alloc_pages().
- don't call soft limit from kswapd and others.

Instead of this, add sysctl like this.

- vm.softlimit_ratio

If vm.softlimit_ratio = 99%,
when sum of all usage of memcg is over 99% of system memory, softlimit runs and reclaim memory until the whole usage will be below 99%.
(or some other trigger can be considered.)

Then,
- We don't have to take care of misc. complicated aspects of memory reclaiming We reclaim memory based on our own logic, then, no influence to global LRU.

I think this approach will hide the all corner case and make merging softlimit to mainline much easier. If you use this approach, RB-tree is the best one to go with (and we don't have to care zone's status.)

Quoted 11 lines

- vm.softlimit_ratio

If vm.softlimit_ratio = 99%,
when sum of all usage of memcg is over 99% of system memory, softlimit runs and reclaim memory until the whole usage will be below 99%.
(or some other trigger can be considered.)

Then,
- We don't have to take care of misc. complicated aspects of memory reclaiming We reclaim memory based on our own logic, then, no influence to global LRU.

Quoted 3 lines

I think this approach will hide the all corner case and make merging softlimit to mainline much easier. If you use this approach, RB-tree is the best one to go with (and we don't have to care zone's status.)

I like the idea in general, but I have concerns about

1. Tracking all cgroup memory, it can quickly get expensive (tracking to check for vm.soft_limit_ratio and for usage)

2. Finding a good default for the sysctl (might not be so hard)

Even today our influence on global LRU is very limited, only when we come under reclaim, we do an additional step of seeing if we can get memory from soft limit groups first.

(1) is a real concern.

Quoted 17 lines · On Tue, 17 Mar 2009 10:10:16 +0530 Balbir Singh <[email protected]>…

Quoted 7 lines

- vm.softlimit_ratio

If vm.softlimit_ratio = 99%,
when sum of all usage of memcg is over 99% of system memory, softlimit runs and reclaim memory until the whole usage will be below 99%.
(or some other trigger can be considered.)

Quoted 7 lines

Then,
- We don't have to take care of misc. complicated aspects of memory reclaiming We reclaim memory based on our own logic, then, no influence to global LRU.

I think this approach will hide the all corner case and make merging softlimit to mainline much easier. If you use this approach, RB-tree is the best one to go with (and we don't have to care zone's status.)

I like the idea in general, but I have concerns about

Quoted 2 lines

1. Tracking all cgroup memory, it can quickly get expensive (tracking to check for vm.soft_limit_ratio and for usage)

Not so expensive because we already tracks them all by default cgroup. Then, what we need is "fast" counter.
Maybe percpu coutner (lib/percpu_counter.c) gives us enough codes for counting.

Checking value ratio is ...how about "once per 1000 increment per cpu" or some ?

Quoted 3 lines

2. Finding a good default for the sysctl (might not be so hard)

I think some parameter like high-low watermark is good and we can find good value as
- low watermak .... max_memory - (sum of all zone->high) * 16 of memory.
- high watermark .... max_memory - (sum_of all zone->high) * 8
(just an example but not so bad.)

Quoted 5 lines

Even today our influence on global LRU is very limited, only when we come under reclaim, we do an additional step of seeing if we can get memory from soft limit groups first.

(1) is a real concern.

Maybe yes. But all memcg will call "charge" "uncharge" codes so, problem is just "counter". I think percpu coutner works enough.

Quoted 23 lines · KAMEZAWA Hiroyuki

Quoted 19 lines · On Tue, 17 Mar 2009 10:10:16 +0530 Balbir Singh <[email protected]>…

Quoted 7 lines

- vm.softlimit_ratio

If vm.softlimit_ratio = 99%,
when sum of all usage of memcg is over 99% of system memory, softlimit runs and reclaim memory until the whole usage will be below 99%.
(or some other trigger can be considered.)

Quoted 7 lines

Then,
- We don't have to take care of misc. complicated aspects of memory reclaiming We reclaim memory based on our own logic, then, no influence to global LRU.

I think this approach will hide the all corner case and make merging softlimit to mainline much easier. If you use this approach, RB-tree is the best one to go with (and we don't have to care zone's status.)

I like the idea in general, but I have concerns about

1. Tracking all cgroup memory, it can quickly get expensive (tracking to check for vm.soft_limit_ratio and for usage)

Quoted 5 lines

Not so expensive because we already tracks them all by default cgroup. Then, what we need is "fast" counter.
Maybe percpu coutner (lib/percpu_counter.c) gives us enough codes for counting.

Checking value ratio is ...how about "once per 1000 increment per cpu" or some ?

That is not true..we don't track them to default cgroup unless memory.use_hiearchy is enabled in the root cgroup.

To do what you
suggest, we have to iterate through all mem cgroups, which is not desirable at all.

Quoted 9 lines

Quoted 3 lines

2. Finding a good default for the sysctl (might not be so hard)

I think some parameter like high-low watermark is good and we can find good value as
- low watermak .... max_memory - (sum of all zone->high) * 16 of memory.
- high watermark .... max_memory - (sum_of all zone->high) * 8
(just an example but not so bad.)

OK..

[offtopic] I liked the per-mem cgroup watermark patches as well. I think we should look at them later on, after soft limits and some other items.

Quoted 4 lines

Quoted 4 lines

Even today our influence on global LRU is very limited, only when we come under reclaim, we do an additional step of seeing if we can get memory from soft limit groups first.

Quoted 5 lines

Quoted 1 line

(1) is a real concern.

Maybe yes. But all memcg will call "charge" "uncharge" codes so, problem is just "counter". I think percpu coutner works enough.

This scheme adds more overhead due to (1), we'll need a global counter and need to protect it, which will serialize all res_counters.

Quoted 29 lines · On Tue, 17 Mar 2009 10:28:50 +0530 Balbir Singh <[email protected]>…

Quoted 27 lines · KAMEZAWA Hiroyuki

Quoted 11 lines · On Tue, 17 Mar 2009 10:10:16 +0530 Balbir Singh <[email protected]>…

Quoted 11 lines

- vm.softlimit_ratio

If vm.softlimit_ratio = 99%,
when sum of all usage of memcg is over 99% of system memory, softlimit runs and reclaim memory until the whole usage will be below 99%.
(or some other trigger can be considered.)

Then,
- We don't have to take care of misc. complicated aspects of memory reclaiming We reclaim memory based on our own logic, then, no influence to global LRU.

Quoted 8 lines

Quoted 3 lines

I think this approach will hide the all corner case and make merging softlimit to mainline much easier. If you use this approach, RB-tree is the best one to go with (and we don't have to care zone's status.)

I like the idea in general, but I have concerns about

1. Tracking all cgroup memory, it can quickly get expensive (tracking to check for vm.soft_limit_ratio and for usage)

Not so expensive because we already tracks them all by default cgroup. Then, what we need is "fast" counter.
Maybe percpu coutner (lib/percpu_counter.c) gives us enough codes for counting.

Quoted 4 lines

Quoted 1 line

Checking value ratio is ...how about "once per 1000 increment per cpu" or some ?

That is not true..we don't track them to default cgroup unless memory.use_hiearchy is enabled in the root cgroup.

What I want to say is "the task which is not attached to user's cgroup is also under defaut cgroup, so we don't need additional hook" Not talking about hierarchy.

Quoted 3 lines

To do what you
suggest, we have to iterate through all mem cgroups, which is not desirable at all.

I don't say percpu counter should be within struct mem_cgroup. DEFINE_PERCPU(overall_usage);
is enough, for example. (just an example.)

Or we already have vmstat (see vmstat.h:: global_page_state())

== si_meminfo()==
val->freeram = global_page_state(NR_FREE_PAGES);
==

It seems we have what we need already. (but usage from this includes usage comes from slab, page table, etc...)

Quoted 10 lines

Quoted 9 lines

Quoted 3 lines

2. Finding a good default for the sysctl (might not be so hard)

I think some parameter like high-low watermark is good and we can find good value as
- low watermak .... max_memory - (sum of all zone->high) * 16 of memory.
- high watermark .... max_memory - (sum_of all zone->high) * 8
(just an example but not so bad.)

Quoted 5 lines

OK..

[offtopic] I liked the per-mem cgroup watermark patches as well. I think we should look at them later on, after soft limits and some other items.

I'm glad if we can reuse logics added by softlimit by per memcg watermarks.

Quoted 6 lines

Quoted 6 lines

Quoted 5 lines

Even today our influence on global LRU is very limited, only when we come under reclaim, we do an additional step of seeing if we can get memory from soft limit groups first.

(1) is a real concern.

Quoted 7 lines

Quoted 3 lines

Maybe yes. But all memcg will call "charge" "uncharge" codes so, problem is just "counter". I think percpu coutner works enough.

This scheme adds more overhead due to (1), we'll need a global counter and need to protect it, which will serialize all res_counters.

It's not necessary. for example, reading vmstat doesn't need global lock.

Quoted 6 lines · KAMEZAWA Hiroyuki

Quoted 2 lines

That is not true..we don't track them to default cgroup unless memory.use_hiearchy is enabled in the root cgroup.

What I want to say is "the task which is not attached to user's cgroup is also under defaut cgroup, so we don't need additional hook" Not talking about hierarchy.

Since all the user pages are tracked in one or the other cgroup, the total accounting is equal to total_lru_pages across all zones/nodes. Your suggestion boils down to if total_lru_pages reaches a threshold, do soft limit reclaim, instead of doing reclaim when there is contention.. right?

Quoted 2 lines

It's not necessary. for example, reading vmstat doesn't need global lock.

It uses atomic values for accounting.

Quoted 8 lines · On Tue, 17 Mar 2009 11:25:06 +0530 Balbir Singh <[email protected]>…

Quoted 6 lines · KAMEZAWA Hiroyuki

Quoted 2 lines

That is not true..we don't track them to default cgroup unless memory.use_hiearchy is enabled in the root cgroup.

What I want to say is "the task which is not attached to user's cgroup is also under defaut cgroup, so we don't need additional hook" Not talking about hierarchy.

Quoted 6 lines

Since all the user pages are tracked in one or the other cgroup, the total accounting is equal to total_lru_pages across all zones/nodes. Your suggestion boils down to if total_lru_pages reaches a threshold, do soft limit reclaim, instead of doing reclaim when there is contention.. right?

Yes.

Quoted 6 lines

Quoted 3 lines

It's not necessary. for example, reading vmstat doesn't need global lock.

It uses atomic values for accounting.

Ah, my point is that "when it comes to usage of global LRU, accounting pages is already done somewhere. we can reuse it." Not means "add some new counter"

Quoted 11 lines · KAMEZAWA Hiroyuki

Quoted 8 lines · On Tue, 17 Mar 2009 11:25:06 +0530 Balbir Singh <[email protected]>…

Quoted 6 lines · KAMEZAWA Hiroyuki

Quoted 2 lines

That is not true..we don't track them to default cgroup unless memory.use_hiearchy is enabled in the root cgroup.

What I want to say is "the task which is not attached to user's cgroup is also under defaut cgroup, so we don't need additional hook" Not talking about hierarchy.

Quoted 8 lines

Quoted 6 lines

Since all the user pages are tracked in one or the other cgroup, the total accounting is equal to total_lru_pages across all zones/nodes. Your suggestion boils down to if total_lru_pages reaches a threshold, do soft limit reclaim, instead of doing reclaim when there is contention.. right?

Yes.

May I suggest that we first do the reclaim on contention and then later on enhance it to add sysctl vm.soft_limit_ratio. I am not proposing the soft limit patches for 2.6.30, but I would like to get them in -mm for wider testing. If in that process the sysctl seems more useful and applicable, we can consider adding it. Adding it right now makes the reclaim logic more complex, having to check if we hit the vm ratio quite often. Do you agree?

Quoted 22 lines · On Tue, 17 Mar 2009 11:52:05 +0530 Balbir Singh <[email protected]>…

Quoted 11 lines · KAMEZAWA Hiroyuki

Quoted 8 lines · On Tue, 17 Mar 2009 11:25:06 +0530 Balbir Singh <[email protected]>…

Quoted 6 lines · KAMEZAWA Hiroyuki

Quoted 2 lines

That is not true..we don't track them to default cgroup unless memory.use_hiearchy is enabled in the root cgroup.

What I want to say is "the task which is not attached to user's cgroup is also under defaut cgroup, so we don't need additional hook" Not talking about hierarchy.

Quoted 8 lines

Quoted 6 lines

Since all the user pages are tracked in one or the other cgroup, the total accounting is equal to total_lru_pages across all zones/nodes. Your suggestion boils down to if total_lru_pages reaches a threshold, do soft limit reclaim, instead of doing reclaim when there is contention.. right?

Yes.

Quoted 8 lines

May I suggest that we first do the reclaim on contention and then later on enhance it to add sysctl vm.soft_limit_ratio. I am not proposing the soft limit patches for 2.6.30, but I would like to get them in -mm for wider testing. If in that process the sysctl seems more useful and applicable, we can consider adding it. Adding it right now makes the reclaim logic more complex, having to check if we hit the vm ratio quite often. Do you agree?

If you can fix zone issues and can answer all Kosaki's request. But you said "this all is not for memory shortage but for softlimit". It seems strange for me to modify memory reclaim path.

Quoted 4 lines

Quoted 4 lines

mm/memcontrol.c +4 -0

+				next_mem =
+					__mem_cgroup_largest_soft_limit_node();
+			} while (next_mem == mem);
+		}

Quoted 8 lines

mm/memcontrol.c int mem_cgroup_resize_memsw_limit(struct mem_cgroup *memcg, · line 2033

Quoted 4 lines

mm/memcontrol.c +4 -0

+		mem->usage_in_excess = res_counter_soft_limit_excess(&mem->res);
+		__mem_cgroup_remove_exceeded(mem);
+		if (mem->usage_in_excess)
+			__mem_cgroup_insert_exceeded(mem);

If next_mem == NULL here, (means "mem" is an only mem_cgroup which excess softlimit.) mem will be found again even if !reclaimed.
plz check.

Yes, We need to add a if (!next_mem) break; Thanks!

Quoted 13 lines · On Mon, 16 Mar 2009 14:05:12 +0530 Balbir Singh <[email protected]>…

Quoted 4 lines

Quoted 4 lines

mm/memcontrol.c +4 -0

+				next_mem =
+					__mem_cgroup_largest_soft_limit_node();
+			} while (next_mem == mem);
+		}

Quoted 8 lines

mm/memcontrol.c int mem_cgroup_resize_memsw_limit(struct mem_cgroup *memcg, · line 2033

Quoted 4 lines

mm/memcontrol.c +4 -0

+		mem->usage_in_excess = res_counter_soft_limit_excess(&mem->res);
+		__mem_cgroup_remove_exceeded(mem);
+		if (mem->usage_in_excess)
+			__mem_cgroup_insert_exceeded(mem);

If next_mem == NULL here, (means "mem" is an only mem_cgroup which excess softlimit.) mem will be found again even if !reclaimed.
plz check.

Quoted 2 lines

Yes, We need to add a if (!next_mem) break; Thanks!

Plz be sure that there can be following case.

1. several memcg is over softlimit.
2. almost all memory usage comes from ANON or tmpfile/shmem.
3. Swapless system
or
Most of memory are mlocked.