Skip to contentSkip to conversation

Memory controller soft limit patches

RFC PATCH In review Balbir Singh 5 participants 19 messages Jan 7, 2009 – Jan 14, 2009

Subsystems (4)
lore
Share & tools
lore
#0 lore raw

cover Memory controller soft limit patches

Here is v1 of the new soft limit implementation. Soft limits is a new feature for the memory resource controller, something similar has existed in the group scheduler in the form of shares. We'll compare shares and soft limits below. I've had soft limit implementations earlier, but I've discarded those approaches in favour of this one.

Soft limits are the most useful feature to have for environments where the administrator wants to overcommit the system, such that only on memory contention do the limits become active. The current soft limits implementation provides a soft_limit_in_bytes interface for the memory controller and not for memory+swap controller. The implementation maintains an RB-Tree of groups that exceed their soft limit and starts reclaiming from the group that exceeds this limit by the maximum amount.

This is an RFC implementation and is not meant for inclusion

TODOs

1. The shares interface is not yet implemented, the current soft limit implementation is not yet hierarchy aware. The end goal is to add a shares interface on top of soft limits and to maintain shares in a manner similar to the group scheduler
2. The current implementation maintains the delta from the soft limit and pushes back groups to their soft limits, a ratio of delta/soft_limit is more useful
3. It would be nice to have more targetted reclaim (in terms of pages to recalim) interface. So that groups are pushed back, close to their soft limits.

Tests
-----

I've run two memory intensive workloads with differing soft limits and seen that they are pushed back to their soft limit on contention. Their usage was their soft limit plus additional memory that they were able to grab on the system.

Please review, comment.

Series
------

memcg-soft-limit-documentation.patch
memcg-add-soft-limit-interface.patch
memcg-organize-over-soft-limit-groups.patch
memcg-soft-limit-reclaim-on-contention.patch

Signature · 7 lines
	Balbir

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
#1 lore raw

RFC PATCH 1/4 Memory controller soft limit documentation

From: Balbir Singh <[email protected]>

Add documentation for soft limit feature support.

---

 Documentation/controllers/memory.txt |   28 ++++++++++++++++++++++++-
 1 file changed, 27 insertions(+), 1 deletion(-)
Documentation/controllers/memory.txt renamed from Documentation/controllers/memory.txt~memcg-soft-limit-documentation +27 -1
diff -puN Documentation/controllers/memory.txt~memcg-soft-limit-documentation Documentation/controllers/memory.txt
@@ -360,7 +360,33 @@ cgroups created below it.
 
 NOTE2: This feature can be enabled/disabled per subtree.
 
-7. TODO
+7. Soft limits
+
+Soft limits allow for greater sharing of memory. The idea behind soft limits
+is to allow control groups to use as much of the memory as needed, provided
+
+a. There is no memory contention
+b. They do not exceed their hard limit
+
+When the system detects memory contention (through do_try_to_free_pages(),
+while allocating), control groups are pushed back to their soft limits if
+possible. If the soft limit of each control group is very high, they are
+pushed back as much as possible to make sure that one control group does not
+starve the others.
+
+7.1 Interface
+
+Soft limits can be setup by using the following commands (in this example we
+assume a soft limit of 256 megabytes)
+
+# echo 256M > memory.soft_limit_in_bytes
+
+If we want to change this to 1G, we can at any time use
+
+# echo 1G > memory.soft_limit_in_bytes
+
+
+8. TODO
 
 1. Add support for accounting huge pages (as a separate controller)
 2. Make per-cgroup scanner reclaim not-shared pages first

_

Signature · 7 lines
	Balbir

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
#4 lore raw

RFC PATCH 2/4 Memory controller soft limit interface

In reply to Balbir Singh

From: Balbir Singh <[email protected]>

Add an interface to allow get/set of soft limits. Soft limits for memory plus swap controller (memsw) is currently not supported. Resource counters have been enhanced to support soft limits and new type RES_SOFT_LIMIT has been added. Unlike hard limits, soft limits can be directly set and do not need any reclaim or checks before setting them to a newer value.

---

 include/linux/res_counter.h |   39 ++++++++++++++++++++++++++++++++++
 kernel/res_counter.c        |    3 ++
 mm/memcontrol.c             |   20 +++++++++++++++++
 3 files changed, 62 insertions(+)
mm/memcontrol.c renamed from mm/memcontrol.c~memcg-add-soft-limit-interface +20 -0
diff -puN mm/memcontrol.c~memcg-add-soft-limit-interface mm/memcontrol.c
@@ -1811,6 +1811,20 @@ static int mem_cgroup_write(struct cgrou
 		else
 			ret = mem_cgroup_resize_memsw_limit(memcg, val);
 		break;
+	case RES_SOFT_LIMIT:
+		ret = res_counter_memparse_write_strategy(buffer, &val);
+		if (ret)
+			break;
+		/*
+		 * For memsw, soft limits are hard to implement in terms
+		 * of semantics, for now, we support soft limits for
+		 * control without swap
+		 */
+		if (type == _MEM)
+			ret = res_counter_set_soft_limit(&memcg->res, val);
+		else
+			ret = -EINVAL;
+		break;
 	default:
 		ret = -EINVAL; /* should be BUG() ? */
 		break;
@@ -2010,6 +2024,12 @@ static struct cftype mem_cgroup_files[] 
 		.read_u64 = mem_cgroup_read,
 	},
 	{
+		.name = "soft_limit_in_bytes",
+		.private = MEMFILE_PRIVATE(_MEM, RES_SOFT_LIMIT),
+		.write_string = mem_cgroup_write,
+		.read_u64 = mem_cgroup_read,
+	},
+	{
 		.name = "failcnt",
 		.private = MEMFILE_PRIVATE(_MEM, RES_FAILCNT),
 		.trigger = mem_cgroup_reset,
include/linux/res_counter.h renamed from include/linux/res_counter.h~memcg-add-soft-limit-interface +39 -0
diff -puN include/linux/res_counter.h~memcg-add-soft-limit-interface include/linux/res_counter.h
@@ -35,6 +35,10 @@ struct res_counter {
 	 */
 	unsigned long long limit;
 	/*
+	 * the limit that usage can be exceed
+	 */
+	unsigned long long soft_limit;
+	/*
 	 * the number of unsuccessful attempts to consume the resource
 	 */
 	unsigned long long failcnt;
@@ -85,6 +89,7 @@ enum {
 	RES_MAX_USAGE,
 	RES_LIMIT,
 	RES_FAILCNT,
+	RES_SOFT_LIMIT,
 };
 
 /*
@@ -130,6 +135,28 @@ static inline bool res_counter_limit_che
 	return false;
 }
 
+/**
+ * Get the difference between the usage and the soft limit
+ * @cnt: The counter
+ *
+ * Returns 0 if usage is less than or equal to soft limit
+ * The difference between usage and soft limit, otherwise.
+ */
+static inline unsigned long long
+res_counter_soft_limit_excess(struct res_counter *cnt)
+{
+	unsigned long long excess;
+	unsigned long flags;
+
+	spin_lock_irqsave(&cnt->lock, flags);
+	if (cnt->usage <= cnt->soft_limit)
+		excess = 0;
+	else
+		excess = cnt->usage - cnt->soft_limit;
+	spin_unlock_irqrestore(&cnt->lock, flags);
+	return excess;
+}
+
 /*
  * Helper function to detect if the cgroup is within it's limit or
  * not. It's currently called from cgroup_rss_prepare()
@@ -178,4 +205,16 @@ static inline int res_counter_set_limit(
 	return ret;
 }
 
+static inline int
+res_counter_set_soft_limit(struct res_counter *cnt,
+				unsigned long long soft_limit)
+{
+	unsigned long flags;
+
+	spin_lock_irqsave(&cnt->lock, flags);
+	cnt->soft_limit = soft_limit;
+	spin_unlock_irqrestore(&cnt->lock, flags);
+	return 0;
+}
+
 #endif
kernel/res_counter.c renamed from kernel/res_counter.c~memcg-add-soft-limit-interface +3 -0
diff -puN kernel/res_counter.c~memcg-add-soft-limit-interface kernel/res_counter.c
@@ -19,6 +19,7 @@ void res_counter_init(struct res_counter
 {
 	spin_lock_init(&counter->lock);
 	counter->limit = (unsigned long long)LLONG_MAX;
+	counter->soft_limit = (unsigned long long)LLONG_MAX;
 	counter->parent = parent;
 }
 
@@ -101,6 +102,8 @@ res_counter_member(struct res_counter *c
 		return &counter->limit;
 	case RES_FAILCNT:
 		return &counter->failcnt;
+	case RES_SOFT_LIMIT:
+		return &counter->soft_limit;
 	};
 
 	BUG();

_

Signature · 7 lines
	Balbir

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
#5 lore raw

RFC PATCH 3/4 Memory controller soft limit organize cgroups

In reply to Balbir Singh

From: Balbir Singh <[email protected]>

This patch introduces a RB-Tree for storing memory cgroups that are over their soft limit. The overall goal is to

1. Add a memory cgroup to the RB-Tree when the soft limit is exceeded. We are careful about updates, updates take place only after a particular time interval has passed
2. We remove the node from the RB-Tree when the usage goes below the soft limit

The next set of patches will exploit the RB-Tree to get the group that is over its soft limit by the largest amount and reclaim from it, when we face memory contention.

---

 mm/memcontrol.c |   78 ++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 78 insertions(+)
mm/memcontrol.c renamed from mm/memcontrol.c~memcg-organize-over-soft-limit-groups +78 -0
diff -puN mm/memcontrol.c~memcg-organize-over-soft-limit-groups mm/memcontrol.c
@@ -28,6 +28,7 @@
 #include <linux/bit_spinlock.h>
 #include <linux/rcupdate.h>
 #include <linux/mutex.h>
+#include <linux/rbtree.h>
 #include <linux/slab.h>
 #include <linux/swap.h>
 #include <linux/spinlock.h>
@@ -119,6 +120,13 @@ struct mem_cgroup_lru_info {
 };
 
 /*
+ * Cgroups above their limits are maintained in a RB-Tree, independent of
+ * their hierarchy representation
+ */
+static struct rb_root mem_cgroup_soft_limit_exceeded_groups;
+static DEFINE_MUTEX(memcg_soft_limit_tree_mutex);
+
+/*
  * The memory controller data structure. The memory controller controls both
  * page cache and RSS per cgroup. We would eventually like to provide
  * statistics based on the statistics developed by Rik Van Riel for clock-pro,
@@ -166,12 +174,18 @@ struct mem_cgroup {
 
 	unsigned int	swappiness;
 
+	struct rb_node mem_cgroup_node;
+	unsigned long long usage_in_excess;
+	unsigned long last_tree_update;
+
 	/*
 	 * statistics. This must be placed at the end of memcg.
 	 */
 	struct mem_cgroup_stat stat;
 };
 
+#define	MEM_CGROUP_TREE_UPDATE_INTERVAL		(HZ)
+
 enum charge_type {
 	MEM_CGROUP_CHARGE_TYPE_CACHE = 0,
 	MEM_CGROUP_CHARGE_TYPE_MAPPED,
@@ -203,6 +217,39 @@ pcg_default_flags[NR_CHARGE_TYPE] = {
 static void mem_cgroup_get(struct mem_cgroup *mem);
 static void mem_cgroup_put(struct mem_cgroup *mem);
 
+static void mem_cgroup_insert_exceeded(struct mem_cgroup *mem)
+{
+	struct rb_node **p = &mem_cgroup_soft_limit_exceeded_groups.rb_node;
+	struct rb_node *parent = NULL;
+	struct mem_cgroup *mem_node;
+
+	mutex_lock(&memcg_soft_limit_tree_mutex);
+	while (*p) {
+		parent = *p;
+		mem_node = rb_entry(parent, struct mem_cgroup, mem_cgroup_node);
+		if (mem->usage_in_excess < mem_node->usage_in_excess)
+			p = &(*p)->rb_left;
+		/*
+		 * We can't avoid mem cgroups that are over their soft
+		 * limit by the same amount
+		 */
+		else if (mem->usage_in_excess >= mem_node->usage_in_excess)
+			p = &(*p)->rb_right;
+	}
+	rb_link_node(&mem->mem_cgroup_node, parent, p);
+	rb_insert_color(&mem->mem_cgroup_node,
+			&mem_cgroup_soft_limit_exceeded_groups);
+	mem->last_tree_update = jiffies;
+	mutex_unlock(&memcg_soft_limit_tree_mutex);
+}
+
+static void mem_cgroup_remove_exceeded(struct mem_cgroup *mem)
+{
+	mutex_lock(&memcg_soft_limit_tree_mutex);
+	rb_erase(&mem->mem_cgroup_node, &mem_cgroup_soft_limit_exceeded_groups);
+	mutex_unlock(&memcg_soft_limit_tree_mutex);
+}
+
 static void mem_cgroup_charge_statistics(struct mem_cgroup *mem,
 					 struct page_cgroup *pc,
 					 bool charge)
@@ -917,6 +964,10 @@ static void __mem_cgroup_commit_charge(s
 				     struct page_cgroup *pc,
 				     enum charge_type ctype)
 {
+	unsigned long long prev_usage_in_excess, new_usage_in_excess;
+	bool updated_tree = false;
+	unsigned long next_update;
+
 	/* try_charge() can return NULL to *memcg, taking care of it. */
 	if (!mem)
 		return;
@@ -937,6 +988,30 @@ static void __mem_cgroup_commit_charge(s
 	mem_cgroup_charge_statistics(mem, pc, true);
 
 	unlock_page_cgroup(pc);
+
+	mem_cgroup_get(mem);
+	prev_usage_in_excess = mem->usage_in_excess;
+	new_usage_in_excess = res_counter_soft_limit_excess(&mem->res);
+
+	next_update = mem->last_tree_update + MEM_CGROUP_TREE_UPDATE_INTERVAL;
+	if (new_usage_in_excess && time_after(jiffies, next_update)) {
+		if (prev_usage_in_excess)
+			mem_cgroup_remove_exceeded(mem);
+		mem_cgroup_insert_exceeded(mem);
+		updated_tree = true;
+	} else if (prev_usage_in_excess && !new_usage_in_excess) {
+		mem_cgroup_remove_exceeded(mem);
+		updated_tree = true;
+	}
+
+	if (updated_tree) {
+		mutex_lock(&memcg_soft_limit_tree_mutex);
+		mem->last_tree_update = jiffies;
+		mem->usage_in_excess = new_usage_in_excess;
+		mutex_unlock(&memcg_soft_limit_tree_mutex);
+	}
+	mem_cgroup_put(mem);
+
 }
 
 /**
@@ -2218,6 +2293,7 @@ mem_cgroup_create(struct cgroup_subsys *
 	if (cont->parent == NULL) {
 		enable_swap_cgroup();
 		parent = NULL;
+		mem_cgroup_soft_limit_exceeded_groups = RB_ROOT;
 	} else {
 		parent = mem_cgroup_from_cont(cont->parent);
 		mem->use_hierarchy = parent->use_hierarchy;
@@ -2231,6 +2307,8 @@ mem_cgroup_create(struct cgroup_subsys *
 		res_counter_init(&mem->memsw, NULL);
 	}
 	mem->last_scanned_child = NULL;
+	mem->usage_in_excess = 0;
+	mem->last_tree_update = 0;	/* Yes, time begins at 0 here */
 	spin_lock_init(&mem->reclaim_param_lock);
 
 	if (parent)

_

Signature · 7 lines
	Balbir

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
#11 lore raw

RFC PATCH 4/4 Memory controller soft limit reclaim on contention

In reply to Balbir Singh

From: Balbir Singh <[email protected]>

This patch allows reclaim from memory cgroups on contention (via the __alloc_pages_internal() path). If a order greater than 0 is specified, we anyway fall back on try_to_free_pages().

memory cgroup soft limit reclaim finds the group that exceeds its soft limit by the largest amount and reclaims pages from it and then reinserts the cgroup into its correct place in the rbtree.

---

 include/linux/memcontrol.h |    1 
 mm/memcontrol.c            |   82 +++++++++++++++++++++++++++++++++--
 mm/page_alloc.c            |   10 +++-
 3 files changed, 88 insertions(+), 5 deletions(-)
mm/memcontrol.c renamed from mm/memcontrol.c~memcg-soft-limit-reclaim-on-contention +78 -4
diff -puN mm/memcontrol.c~memcg-soft-limit-reclaim-on-contention mm/memcontrol.c
@@ -177,6 +177,7 @@ struct mem_cgroup {
 	struct rb_node mem_cgroup_node;
 	unsigned long long usage_in_excess;
 	unsigned long last_tree_update;
+	bool on_tree;
 
 	/*
 	 * statistics. This must be placed at the end of memcg.
@@ -184,7 +185,7 @@ struct mem_cgroup {
 	struct mem_cgroup_stat stat;
 };
 
-#define	MEM_CGROUP_TREE_UPDATE_INTERVAL		(HZ)
+#define	MEM_CGROUP_TREE_UPDATE_INTERVAL		(HZ/4)
 
 enum charge_type {
 	MEM_CGROUP_CHARGE_TYPE_CACHE = 0,
@@ -217,13 +218,15 @@ pcg_default_flags[NR_CHARGE_TYPE] = {
 static void mem_cgroup_get(struct mem_cgroup *mem);
 static void mem_cgroup_put(struct mem_cgroup *mem);
 
-static void mem_cgroup_insert_exceeded(struct mem_cgroup *mem)
+static void __mem_cgroup_insert_exceeded(struct mem_cgroup *mem)
 {
 	struct rb_node **p = &mem_cgroup_soft_limit_exceeded_groups.rb_node;
 	struct rb_node *parent = NULL;
 	struct mem_cgroup *mem_node;
 
-	mutex_lock(&memcg_soft_limit_tree_mutex);
+	if (mem->on_tree)
+		return;
+
 	while (*p) {
 		parent = *p;
 		mem_node = rb_entry(parent, struct mem_cgroup, mem_cgroup_node);
@@ -240,16 +243,54 @@ static void mem_cgroup_insert_exceeded(s
 	rb_insert_color(&mem->mem_cgroup_node,
 			&mem_cgroup_soft_limit_exceeded_groups);
 	mem->last_tree_update = jiffies;
+	mem->on_tree = true;
+}
+
+static void __mem_cgroup_remove_exceeded(struct mem_cgroup *mem)
+{
+	if (!mem->on_tree)
+		return;
+	rb_erase(&mem->mem_cgroup_node, &mem_cgroup_soft_limit_exceeded_groups);
+	mem->on_tree = false;
+}
+
+static void mem_cgroup_insert_exceeded(struct mem_cgroup *mem)
+{
+	mutex_lock(&memcg_soft_limit_tree_mutex);
+	__mem_cgroup_insert_exceeded(mem);
 	mutex_unlock(&memcg_soft_limit_tree_mutex);
 }
 
 static void mem_cgroup_remove_exceeded(struct mem_cgroup *mem)
 {
 	mutex_lock(&memcg_soft_limit_tree_mutex);
-	rb_erase(&mem->mem_cgroup_node, &mem_cgroup_soft_limit_exceeded_groups);
+	__mem_cgroup_remove_exceeded(mem);
 	mutex_unlock(&memcg_soft_limit_tree_mutex);
 }
 
+static struct mem_cgroup *mem_cgroup_get_largest_soft_limit_exceeding_node(void)
+{
+	struct rb_node *rightmost = NULL;
+	struct mem_cgroup *mem = NULL;
+
+	mutex_lock(&memcg_soft_limit_tree_mutex);
+	rightmost = rb_last(&mem_cgroup_soft_limit_exceeded_groups);
+	if (!rightmost)
+		goto done;		/* Nothing to reclaim from */
+
+	mem = rb_entry(rightmost, struct mem_cgroup, mem_cgroup_node);
+	mem_cgroup_get(mem);
+	/*
+	 * Remove the node now but someone else can add it back,
+	 * we will to add it back at the end of reclaim to its correct
+	 * position in the tree.
+	 */
+	__mem_cgroup_remove_exceeded(mem);
+done:
+	mutex_unlock(&memcg_soft_limit_tree_mutex);
+	return mem;
+}
+
 static void mem_cgroup_charge_statistics(struct mem_cgroup *mem,
 					 struct page_cgroup *pc,
 					 bool charge)
@@ -1795,6 +1836,37 @@ try_to_free:
 	goto out;
 }
 
+unsigned long mem_cgroup_soft_limit_reclaim(gfp_t gfp_mask)
+{
+	unsigned long nr_reclaimed = 0;
+	struct mem_cgroup *mem;
+
+	do {
+		mem = mem_cgroup_get_largest_soft_limit_exceeding_node();
+		if (!mem)
+			break;
+		if (mem_cgroup_is_obsolete(mem)) {
+			mem_cgroup_put(mem);
+			continue;
+		}
+		nr_reclaimed +=
+			try_to_free_mem_cgroup_pages(mem, gfp_mask, false,
+							get_swappiness(mem));
+		mutex_lock(&memcg_soft_limit_tree_mutex);
+		mem->usage_in_excess = res_counter_soft_limit_excess(&mem->res);
+		/*
+		 * We need to remove and reinsert the node in its correct
+		 * position
+		 */
+		__mem_cgroup_remove_exceeded(mem);
+		if (mem->usage_in_excess)
+			__mem_cgroup_insert_exceeded(mem);
+		mutex_unlock(&memcg_soft_limit_tree_mutex);
+		mem_cgroup_put(mem);
+	} while (!nr_reclaimed);
+	return nr_reclaimed;
+}
+
 int mem_cgroup_force_empty_write(struct cgroup *cont, unsigned int event)
 {
 	return mem_cgroup_force_empty(mem_cgroup_from_cont(cont), true);
@@ -2309,6 +2381,8 @@ mem_cgroup_create(struct cgroup_subsys *
 	mem->last_scanned_child = NULL;
 	mem->usage_in_excess = 0;
 	mem->last_tree_update = 0;	/* Yes, time begins at 0 here */
+	mem->on_tree = false;
+
 	spin_lock_init(&mem->reclaim_param_lock);
 
 	if (parent)

diff -puN mm/vmscan.c~memcg-soft-limit-reclaim-on-contention mm/vmscan.c

include/linux/memcontrol.h renamed from include/linux/memcontrol.h~memcg-soft-limit-reclaim-on-contention +1 -0
diff -puN include/linux/memcontrol.h~memcg-soft-limit-reclaim-on-contention include/linux/memcontrol.h
@@ -117,6 +117,7 @@ static inline bool mem_cgroup_disabled(v
 }
 
 extern bool mem_cgroup_oom_called(struct task_struct *task);
+extern unsigned long mem_cgroup_soft_limit_reclaim(gfp_t gfp_mask);
 
 #else /* CONFIG_CGROUP_MEM_RES_CTLR */
 struct mem_cgroup;
mm/page_alloc.c renamed from mm/page_alloc.c~memcg-soft-limit-reclaim-on-contention +9 -1
diff -puN mm/page_alloc.c~memcg-soft-limit-reclaim-on-contention mm/page_alloc.c
@@ -1582,7 +1582,15 @@ nofail_alloc:
 	reclaim_state.reclaimed_slab = 0;
 	p->reclaim_state = &reclaim_state;
 
-	did_some_progress = try_to_free_pages(zonelist, order, gfp_mask);
+	did_some_progress = mem_cgroup_soft_limit_reclaim(gfp_mask);
+	/*
+	 * If we made no progress or need higher order allocations
+	 * try_to_free_pages() is still our best bet, since mem_cgroup
+	 * reclaim does not handle freeing pages greater than order 0
+	 */
+	if (!did_some_progress || order)
+		did_some_progress = try_to_free_pages(zonelist, order,
+							gfp_mask);
 
 	p->reclaim_state = NULL;
 	p->flags &= ~PF_MEMALLOC;

_

Signature · 1 line
	Balbir
#12 lore raw
In reply to Balbir Singh
Quoted 15 lines · Balbir Singh

Here is v1 of the new soft limit implementation. Soft limits is a new feature for the memory resource controller, something similar has existed in the group scheduler in the form of shares. We'll compare shares and soft limits below. I've had soft limit implementations earlier, but I've discarded those approaches in favour of this one.

Soft limits are the most useful feature to have for environments where the administrator wants to overcommit the system, such that only on memory contention do the limits become active. The current soft limits implementation provides a soft_limit_in_bytes interface for the memory controller and not for memory+swap controller. The implementation maintains an RB-Tree of groups that exceed their soft limit and starts reclaiming from the group that exceeds this limit by the maximum amount.

Quoted 8 lines

This is an RFC implementation and is not meant for inclusion

TODOs

1. The shares interface is not yet implemented, the current soft limit implementation is not yet hierarchy aware. The end goal is to add a shares interface on top of soft limits and to maintain shares in a manner similar to the group scheduler

Just to clarify, when there is no contention, you want to share memory proportionally?

thanks,

Signature · 8 lines
regards,
Dhaval

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
#15 lore raw
In reply to Balbir Singh
Quoted 15 lines · On Thu, 08 Jan 2009 00:11:10 +0530 Balbir Singh <[email protected]>…

Here is v1 of the new soft limit implementation. Soft limits is a new feature for the memory resource controller, something similar has existed in the group scheduler in the form of shares. We'll compare shares and soft limits below. I've had soft limit implementations earlier, but I've discarded those approaches in favour of this one.

Soft limits are the most useful feature to have for environments where the administrator wants to overcommit the system, such that only on memory contention do the limits become active. The current soft limits implementation provides a soft_limit_in_bytes interface for the memory controller and not for memory+swap controller. The implementation maintains an RB-Tree of groups that exceed their soft limit and starts reclaiming from the group that exceeds this limit by the maximum amount.

Quoted 2 lines

This is an RFC implementation and is not meant for inclusion

Core implemantation seems simple and the feature sounds good. But, before reviewing into details, 3 points.

1. please fix current bugs on hierarchy management, before new feature. AFAIK, OOM-Kill under hierarchy is broken. (I have patches but waits for merge window close.)
I wonder there will be some others. Lockdep error which Nishimura reported are all fixed now ?

2. You inserts reclaim-by-soft-limit into alloc_pages(). But, to do this, you have to pass zonelist to try_to_free_mem_cgroup_pages() and have to modify try_to_free_mem_cgroup_pages().
2-a) If not, when the memory request is for gfp_mask==GFP_DMA or allocation
is under a cpuset, memory reclaim will not work correctlly.
2-b) try_to_free_mem_cgroup_pages() cannot do good work for order > 1 allocation.

Please try fake-numa (or real NUMA machine) and cpuset.

3. If you want to insert hooks to "generic" page allocator, it's better to add CC to Rik van Riel, Kosaki Motohiro, at leaset.

To be honest, I myself don't like to add a hook to alloc_pages() directly. Can we implment call soft-limit like kswapd (or on kswapd()) ? i.e. in moderate way ?

A happy new year,

-Kame

Quoted 23 lines

TODOs

1. The shares interface is not yet implemented, the current soft limit implementation is not yet hierarchy aware. The end goal is to add a shares interface on top of soft limits and to maintain shares in a manner similar to the group scheduler
2. The current implementation maintains the delta from the soft limit and pushes back groups to their soft limits, a ratio of delta/soft_limit is more useful
3. It would be nice to have more targetted reclaim (in terms of pages to recalim) interface. So that groups are pushed back, close to their soft limits.

Tests
-----

I've run two memory intensive workloads with differing soft limits and seen that they are pushed back to their soft limit on contention. Their usage was their soft limit plus additional memory that they were able to grab on the system.

Please review, comment.

Quoted 11 lines

Series
------

memcg-soft-limit-documentation.patch
memcg-add-soft-limit-interface.patch
memcg-organize-over-soft-limit-groups.patch
memcg-soft-limit-reclaim-on-contention.patch

Signature · 1 line
	Balbir

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#13 lore raw
In reply to Dhaval Giani
Quoted 20 lines · On Thu, 8 Jan 2009 00:26:27 +0530 Dhaval Giani <[email protected]>…
Quoted 19 lines · Balbir Singh

Here is v1 of the new soft limit implementation. Soft limits is a new feature for the memory resource controller, something similar has existed in the group scheduler in the form of shares. We'll compare shares and soft limits below. I've had soft limit implementations earlier, but I've discarded those approaches in favour of this one.

Soft limits are the most useful feature to have for environments where the administrator wants to overcommit the system, such that only on memory contention do the limits become active. The current soft limits implementation provides a soft_limit_in_bytes interface for the memory controller and not for memory+swap controller. The implementation maintains an RB-Tree of groups that exceed their soft limit and starts reclaiming from the group that exceeds this limit by the maximum amount.

This is an RFC implementation and is not meant for inclusion

TODOs

Quoted 8 lines
Quoted 4 lines

1. The shares interface is not yet implemented, the current soft limit implementation is not yet hierarchy aware. The end goal is to add a shares interface on top of soft limits and to maintain shares in a manner similar to the group scheduler

Just to clarify, when there is no contention, you want to share memory proportionally?

I don't like to add "share" as the kernel interface of memcg. We used "bytes" to do (hard) limit. Please just use "bytes".

Thanks,
-Kame

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#6 lore raw

Re: [RFC][PATCH 3/4] Memory controller soft limit organize cgroups

In reply to Balbir Singh
Quoted 11 lines · On Thu, 08 Jan 2009 00:11:28 +0530 Balbir Singh <[email protected]>…

From: Balbir Singh <[email protected]>

This patch introduces a RB-Tree for storing memory cgroups that are over their soft limit. The overall goal is to

1. Add a memory cgroup to the RB-Tree when the soft limit is exceeded. We are careful about updates, updates take place only after a particular time interval has passed
2. We remove the node from the RB-Tree when the usage goes below the soft limit

Quoted 4 lines

The next set of patches will exploit the RB-Tree to get the group that is over its soft limit by the largest amount and reclaim from it, when we face memory contention.

Hmm, Could you clarify following ?

- Usage of memory at insertsion and usage of memory at reclaim is different. So, this *sorted* order by RB-tree isn't the best order in general. Why don't you sort this at memory-reclaim dynamically ?
- Considering above, the look of RB tree can be

                +30M (an amount over soft limit is 30M)
                /  \
             -15M   +60M
     ?

    At least, pleease remove the node at uncharge() when the usage goes down.

Thanks,
-Kame

Quoted 149 lines

---

 mm/memcontrol.c |   78 ++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 78 insertions(+)

diff -puN mm/memcontrol.c~memcg-organize-over-soft-limit-groups mm/memcontrol.c

mm/memcontrol.c renamed from mm/memcontrol.c~memcg-organize-over-soft-limit-groups +78 -0
@@ -28,6 +28,7 @@
 #include <linux/bit_spinlock.h>
 #include <linux/rcupdate.h>
 #include <linux/mutex.h>
+#include <linux/rbtree.h>
 #include <linux/slab.h>
 #include <linux/swap.h>
 #include <linux/spinlock.h>
@@ -119,6 +120,13 @@ struct mem_cgroup_lru_info {
 };
 
 /*
+ * Cgroups above their limits are maintained in a RB-Tree, independent of
+ * their hierarchy representation
+ */
+static struct rb_root mem_cgroup_soft_limit_exceeded_groups;
+static DEFINE_MUTEX(memcg_soft_limit_tree_mutex);
+
+/*
  * The memory controller data structure. The memory controller controls both
  * page cache and RSS per cgroup. We would eventually like to provide
  * statistics based on the statistics developed by Rik Van Riel for clock-pro,
@@ -166,12 +174,18 @@ struct mem_cgroup {
 
 	unsigned int	swappiness;
 
+	struct rb_node mem_cgroup_node;
+	unsigned long long usage_in_excess;
+	unsigned long last_tree_update;
+
 	/*
 	 * statistics. This must be placed at the end of memcg.
 	 */
 	struct mem_cgroup_stat stat;
 };
 
+#define	MEM_CGROUP_TREE_UPDATE_INTERVAL		(HZ)
+
 enum charge_type {
 	MEM_CGROUP_CHARGE_TYPE_CACHE = 0,
 	MEM_CGROUP_CHARGE_TYPE_MAPPED,
@@ -203,6 +217,39 @@ pcg_default_flags[NR_CHARGE_TYPE] = {
 static void mem_cgroup_get(struct mem_cgroup *mem);
 static void mem_cgroup_put(struct mem_cgroup *mem);
 
+static void mem_cgroup_insert_exceeded(struct mem_cgroup *mem)
+{
+	struct rb_node **p = &mem_cgroup_soft_limit_exceeded_groups.rb_node;
+	struct rb_node *parent = NULL;
+	struct mem_cgroup *mem_node;
+
+	mutex_lock(&memcg_soft_limit_tree_mutex);
+	while (*p) {
+		parent = *p;
+		mem_node = rb_entry(parent, struct mem_cgroup, mem_cgroup_node);
+		if (mem->usage_in_excess < mem_node->usage_in_excess)
+			p = &(*p)->rb_left;
+		/*
+		 * We can't avoid mem cgroups that are over their soft
+		 * limit by the same amount
+		 */
+		else if (mem->usage_in_excess >= mem_node->usage_in_excess)
+			p = &(*p)->rb_right;
+	}
+	rb_link_node(&mem->mem_cgroup_node, parent, p);
+	rb_insert_color(&mem->mem_cgroup_node,
+			&mem_cgroup_soft_limit_exceeded_groups);
+	mem->last_tree_update = jiffies;
+	mutex_unlock(&memcg_soft_limit_tree_mutex);
+}
+
+static void mem_cgroup_remove_exceeded(struct mem_cgroup *mem)
+{
+	mutex_lock(&memcg_soft_limit_tree_mutex);
+	rb_erase(&mem->mem_cgroup_node, &mem_cgroup_soft_limit_exceeded_groups);
+	mutex_unlock(&memcg_soft_limit_tree_mutex);
+}
+
 static void mem_cgroup_charge_statistics(struct mem_cgroup *mem,
 					 struct page_cgroup *pc,
 					 bool charge)
@@ -917,6 +964,10 @@ static void __mem_cgroup_commit_charge(s
 				     struct page_cgroup *pc,
 				     enum charge_type ctype)
 {
+	unsigned long long prev_usage_in_excess, new_usage_in_excess;
+	bool updated_tree = false;
+	unsigned long next_update;
+
 	/* try_charge() can return NULL to *memcg, taking care of it. */
 	if (!mem)
 		return;
@@ -937,6 +988,30 @@ static void __mem_cgroup_commit_charge(s
 	mem_cgroup_charge_statistics(mem, pc, true);
 
 	unlock_page_cgroup(pc);
+
+	mem_cgroup_get(mem);
+	prev_usage_in_excess = mem->usage_in_excess;
+	new_usage_in_excess = res_counter_soft_limit_excess(&mem->res);
+
+	next_update = mem->last_tree_update + MEM_CGROUP_TREE_UPDATE_INTERVAL;
+	if (new_usage_in_excess && time_after(jiffies, next_update)) {
+		if (prev_usage_in_excess)
+			mem_cgroup_remove_exceeded(mem);
+		mem_cgroup_insert_exceeded(mem);
+		updated_tree = true;
+	} else if (prev_usage_in_excess && !new_usage_in_excess) {
+		mem_cgroup_remove_exceeded(mem);
+		updated_tree = true;
+	}
+
+	if (updated_tree) {
+		mutex_lock(&memcg_soft_limit_tree_mutex);
+		mem->last_tree_update = jiffies;
+		mem->usage_in_excess = new_usage_in_excess;
+		mutex_unlock(&memcg_soft_limit_tree_mutex);
+	}
+	mem_cgroup_put(mem);
+
 }
 
 /**
@@ -2218,6 +2293,7 @@ mem_cgroup_create(struct cgroup_subsys *
 	if (cont->parent == NULL) {
 		enable_swap_cgroup();
 		parent = NULL;
+		mem_cgroup_soft_limit_exceeded_groups = RB_ROOT;
 	} else {
 		parent = mem_cgroup_from_cont(cont->parent);
 		mem->use_hierarchy = parent->use_hierarchy;
@@ -2231,6 +2307,8 @@ mem_cgroup_create(struct cgroup_subsys *
 		res_counter_init(&mem->memsw, NULL);
 	}
 	mem->last_scanned_child = NULL;
+	mem->usage_in_excess = 0;
+	mem->last_tree_update = 0;	/* Yes, time begins at 0 here */
 	spin_lock_init(&mem->reclaim_param_lock);
 
 	if (parent)
Quoted 10 lines

_

Signature · 6 lines
	Balbir
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#14 lore raw
In reply to KAMEZAWA Hiroyuki
Quoted 28 lines · KAMEZAWA Hiroyuki
Quoted 25 lines · On Thu, 8 Jan 2009 00:26:27 +0530 Dhaval Giani <[email protected]>…
Quoted 23 lines · Balbir Singh

Here is v1 of the new soft limit implementation. Soft limits is a new feature for the memory resource controller, something similar has existed in the group scheduler in the form of shares. We'll compare shares and soft limits below. I've had soft limit implementations earlier, but I've discarded those approaches in favour of this one.

Soft limits are the most useful feature to have for environments where the administrator wants to overcommit the system, such that only on memory contention do the limits become active. The current soft limits implementation provides a soft_limit_in_bytes interface for the memory controller and not for memory+swap controller. The implementation maintains an RB-Tree of groups that exceed their soft limit and starts reclaiming from the group that exceeds this limit by the maximum amount.

This is an RFC implementation and is not meant for inclusion

TODOs

1. The shares interface is not yet implemented, the current soft limit implementation is not yet hierarchy aware. The end goal is to add a shares interface on top of soft limits and to maintain shares in a manner similar to the group scheduler

Quoted 6 lines
Quoted 3 lines

Just to clarify, when there is no contention, you want to share memory proportionally?

I don't like to add "share" as the kernel interface of memcg. We used "bytes" to do (hard) limit. Please just use "bytes".

Yes, we'll have soft limit in bytes, but for a hierarchical view, shares do make a lot of sense. The user can use whichever interface suits them the most.

Signature · 7 lines
	Balbir

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
#16 lore raw
In reply to KAMEZAWA Hiroyuki
Quoted 18 lines · KAMEZAWA Hiroyuki
Quoted 15 lines · On Thu, 08 Jan 2009 00:11:10 +0530 Balbir Singh <[email protected]>…

Here is v1 of the new soft limit implementation. Soft limits is a new feature for the memory resource controller, something similar has existed in the group scheduler in the form of shares. We'll compare shares and soft limits below. I've had soft limit implementations earlier, but I've discarded those approaches in favour of this one.

Soft limits are the most useful feature to have for environments where the administrator wants to overcommit the system, such that only on memory contention do the limits become active. The current soft limits implementation provides a soft_limit_in_bytes interface for the memory controller and not for memory+swap controller. The implementation maintains an RB-Tree of groups that exceed their soft limit and starts reclaiming from the group that exceeds this limit by the maximum amount.

Quoted 3 lines
Quoted 2 lines

This is an RFC implementation and is not meant for inclusion

Core implemantation seems simple and the feature sounds good.

Thanks!

Quoted 5 lines

But, before reviewing into details, 3 points.

1. please fix current bugs on hierarchy management, before new feature. AFAIK, OOM-Kill under hierarchy is broken. (I have patches but waits for merge window close.)

I've not hit the OOM-kill issue under hierarchy so far, is the OOM killer selecting a bad task to kill? I'll debug/reproduce the issue. I am not posting these patches for inclusion, fixing bugs is definitely the highest priority.

Quoted 2 lines

I wonder there will be some others. Lockdep error which Nishimura reported are all fixed now ?

I run all my kernels and tests with lockdep enabled, I did not see any lockdep errors showing up.

Quoted 6 lines

2. You inserts reclaim-by-soft-limit into alloc_pages(). But, to do this, you have to pass zonelist to try_to_free_mem_cgroup_pages() and have to modify try_to_free_mem_cgroup_pages().
2-a) If not, when the memory request is for gfp_mask==GFP_DMA or allocation
is under a cpuset, memory reclaim will not work correctlly.

The idea behind adding the code in alloc_pages() is to detect contention and trim mem cgroups down, if they have grown beyond their soft limit

Quoted 3 lines

2-b) try_to_free_mem_cgroup_pages() cannot do good work for order > 1 allocation.

Please try fake-numa (or real NUMA machine) and cpuset.

Yes, order > 1 is documented in the patch and you can see the code as well. Your suggestion is to look at the gfp_mask as well, I'll do that.

Quoted 3 lines

3. If you want to insert hooks to "generic" page allocator, it's better to add CC to Rik van Riel, Kosaki Motohiro, at leaset.

Sure, I'll do that in the next patchset.

Quoted 5 lines

To be honest, I myself don't like to add a hook to alloc_pages() directly. Can we implment call soft-limit like kswapd (or on kswapd()) ? i.e. in moderate way ?

Yes, that might be another point to experiment with, I'll try that in the next iteration.

Quoted 2 lines

A happy new year,

A very happy new year to you as well.

Quoted 2 lines

-Kame

Signature · 7 lines
	Balbir

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
#17 lore raw
Quoted 28 lines · On Thu, 8 Jan 2009 09:29:30 +0530 Balbir Singh <[email protected]>…
Quoted 18 lines · KAMEZAWA Hiroyuki
Quoted 15 lines · On Thu, 08 Jan 2009 00:11:10 +0530 Balbir Singh <[email protected]>…

Here is v1 of the new soft limit implementation. Soft limits is a new feature for the memory resource controller, something similar has existed in the group scheduler in the form of shares. We'll compare shares and soft limits below. I've had soft limit implementations earlier, but I've discarded those approaches in favour of this one.

Soft limits are the most useful feature to have for environments where the administrator wants to overcommit the system, such that only on memory contention do the limits become active. The current soft limits implementation provides a soft_limit_in_bytes interface for the memory controller and not for memory+swap controller. The implementation maintains an RB-Tree of groups that exceed their soft limit and starts reclaiming from the group that exceeds this limit by the maximum amount.

Quoted 3 lines
Quoted 2 lines

This is an RFC implementation and is not meant for inclusion

Core implemantation seems simple and the feature sounds good.

Thanks!

Quoted 2 lines

But, before reviewing into details, 3 points.

Quoted 9 lines
Quoted 3 lines

1. please fix current bugs on hierarchy management, before new feature. AFAIK, OOM-Kill under hierarchy is broken. (I have patches but waits for merge window close.)

I've not hit the OOM-kill issue under hierarchy so far, is the OOM killer selecting a bad task to kill? I'll debug/reproduce the issue. I am not posting these patches for inclusion, fixing bugs is definitely the highest priority.

Assume follwoing hierarchy.

group_A/ limit=100M usage=1M
group_01/ no limit usage=1M
group_02/ no limit usage=98M (does memory leak.)

Q. What happens a task on group_02 causes oom ? A. A task in group_A dies.

is my problem. (As I said, I'll post a patch .) This is my homework for a month. (I'll use CSS_ID to fix this.)
Any this will allow to skip my logic to check "Is this OOM is from memcg?" And makes system panic if vm.panic_on_oom==1.

Quoted 6 lines
Quoted 2 lines

I wonder there will be some others. Lockdep error which Nishimura reported are all fixed now ?

I run all my kernels and tests with lockdep enabled, I did not see any lockdep errors showing up.

ok.

Quoted 11 lines
Quoted 6 lines

2. You inserts reclaim-by-soft-limit into alloc_pages(). But, to do this, you have to pass zonelist to try_to_free_mem_cgroup_pages() and have to modify try_to_free_mem_cgroup_pages().
2-a) If not, when the memory request is for gfp_mask==GFP_DMA or allocation
is under a cpuset, memory reclaim will not work correctlly.

The idea behind adding the code in alloc_pages() is to detect contention and trim mem cgroups down, if they have grown beyond their soft limit

Allowing usual direct reclaim go on and just waking up "balance_soft_limit_daemon()" will be enough.

Quoted 8 lines
Quoted 3 lines

2-b) try_to_free_mem_cgroup_pages() cannot do good work for order > 1 allocation.

Please try fake-numa (or real NUMA machine) and cpuset.

Yes, order > 1 is documented in the patch and you can see the code as well. Your suggestion is to look at the gfp_mask as well, I'll do that.

and zonelist/nodemask.

generic try_to_free_pages() doesn't have nodemask as its argument but it checks cpuset.

In shrink_zones().
==
1504 /*

1505                  * Take care memory controller reclaiming has small influence
1506                  * to global LRU.
1507                  */

1508 if (scan_global_lru(sc)) {
1509 if (!cpuset_zone_allowed_hardwall(zone, GFP_KERNEL))
1510 continue;
1511 note_zone_scanning_priority(zone, priority);
1512
1513 if (zone_is_all_unreclaimable(zone) &&
1514 priority != DEF_PRIORITY)
1515 continue; /* Let kswapd poll it */
1516 sc->all_unreclaimable = 0;
1517 } else {
1518 /*

1519                          * Ignore cpuset limitation here. We just want to reduce
1520                          * # of used pages by us regardless of memory shortage.
1521                          */

1522 sc->all_unreclaimable = 0;
1523 mem_cgroup_note_reclaim_priority(sc->mem_cgroup,
1524 priority);
1525 }
==
This is because "reclaim by memcg" can happen even if there are enough memory. try_to_free_mem_cgroup_pages() is called when "hit limit".

So, there will be some issues to be improved if you want to use try_to_free_mem_cgroup_pages() for recovering "memory shortage". I think above is one of issue. Some more assumption will corrupt.

-Kame

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#7 lore raw

Re: [RFC][PATCH 3/4] Memory controller soft limit organize cgroups

In reply to KAMEZAWA Hiroyuki
Quoted 19 lines · KAMEZAWA Hiroyuki
Quoted 11 lines · On Thu, 08 Jan 2009 00:11:28 +0530 Balbir Singh <[email protected]>…

From: Balbir Singh <[email protected]>

This patch introduces a RB-Tree for storing memory cgroups that are over their soft limit. The overall goal is to

1. Add a memory cgroup to the RB-Tree when the soft limit is exceeded. We are careful about updates, updates take place only after a particular time interval has passed
2. We remove the node from the RB-Tree when the usage goes below the soft limit

Quoted 4 lines

The next set of patches will exploit the RB-Tree to get the group that is over its soft limit by the largest amount and reclaim from it, when we face memory contention.

Quoted 4 lines

Hmm, Could you clarify following ?

- Usage of memory at insertsion and usage of memory at reclaim is different. So, this *sorted* order by RB-tree isn't the best order in general.

True, but we frequently update the tree at an interval of HZ/4. Updating at every page fault sounded like an overkill and building the entire tree at reclaim is an overkill too.

Quoted 6 lines

Why don't you sort this at memory-reclaim dynamically ?
- Considering above, the look of RB tree can be

                +30M (an amount over soft limit is 30M)
                /  \
             -15M   +60M

We don't have elements below their soft limit in the tree

Quoted 4 lines

?

At least, pleease remove the node at uncharge() when the usage goes down.

We do remove the tree if it goes under its soft limit at commit_charge, I thought I had the same code in uncharge(), but clearly that is missing. Thanks, I'll add it there.

Quoted 2 lines

Thanks,
-Kame

Signature · 7 lines
	Balbir

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
#8 lore raw

Re: [RFC][PATCH 3/4] Memory controller soft limit organize cgroups

Quoted 10 lines · On Thu, 8 Jan 2009 09:55:58 +0530 Balbir Singh <[email protected]>…
Quoted 4 lines · KAMEZAWA Hiroyuki

Hmm, Could you clarify following ?

- Usage of memory at insertsion and usage of memory at reclaim is different. So, this *sorted* order by RB-tree isn't the best order in general.

True, but we frequently update the tree at an interval of HZ/4. Updating at every page fault sounded like an overkill and building the entire tree at reclaim is an overkill too.

"sort" is not necessary.
If this feature is implemented as background daemon, just select the worst one at each iteration is enough.

Quoted 9 lines
Quoted 6 lines

Why don't you sort this at memory-reclaim dynamically ?
- Considering above, the look of RB tree can be

                +30M (an amount over soft limit is 30M)
                /  \
             -15M   +60M

We don't have elements below their soft limit in the tree

Quoted 9 lines
Quoted 4 lines

?

At least, pleease remove the node at uncharge() when the usage goes down.

We do remove the tree if it goes under its soft limit at commit_charge, I thought I had the same code in uncharge(), but clearly that is missing. Thanks, I'll add it there.

Ah, ok. I missed it. Thank you for clalification.

Regards,
-Kame

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#9 lore raw

Re: [RFC][PATCH 3/4] Memory controller soft limit organize cgroups

Quoted 9 lines · KAMEZAWA Hiroyuki
Quoted 6 lines · On Thu, 8 Jan 2009 09:55:58 +0530 Balbir Singh <[email protected]>…
Quoted 4 lines · KAMEZAWA Hiroyuki

Hmm, Could you clarify following ?

- Usage of memory at insertsion and usage of memory at reclaim is different. So, this *sorted* order by RB-tree isn't the best order in general.

Quoted 7 lines
Quoted 4 lines

True, but we frequently update the tree at an interval of HZ/4. Updating at every page fault sounded like an overkill and building the entire tree at reclaim is an overkill too.

"sort" is not necessary.
If this feature is implemented as background daemon, just select the worst one at each iteration is enough.

OK, definitely an alternative worth considering, but the trade-off is lazy building (your suggestion), which involves actively seeing the usage of all cgroups (and if they are large, O(c), c is number of cgroups can be quite a bit) versus building the tree as and when the fault occurs and controlled by some interval.

Signature · 7 lines
	Balbir

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
#18 lore raw
In reply to Balbir Singh
Quoted 9 lines
Quoted 3 lines

1. please fix current bugs on hierarchy management, before new feature. AFAIK, OOM-Kill under hierarchy is broken. (I have patches but waits for merge window close.)

I've not hit the OOM-kill issue under hierarchy so far, is the OOM killer selecting a bad task to kill? I'll debug/reproduce the issue. I am not posting these patches for inclusion, fixing bugs is definitely the highest priority.

I agree.

Just FYI, I have several bug fix patches for current memcg(that is for .29). I've been testing them now, and it survives my test(rmdir aftre task move under memory pressure and page migration) w/o big problem(except oom) for hours in both use_hierarchy==0/1 case.

Quoted 6 lines
Quoted 2 lines

I wonder there will be some others. Lockdep error which Nishimura reported are all fixed now ?

I run all my kernels and tests with lockdep enabled, I did not see any lockdep errors showing up.

I think Paul's hierarchy_mutex patches fixed the dead lock, I haven't seen the dead lock after the patch.
(Although, it may cause another dead lock when other subsystems are added.)

Thanks,
Daisuke Nishimura.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#10 lore raw

Re: [RFC][PATCH 3/4] Memory controller soft limit organize cgroups

In reply to Balbir Singh
Quoted 19 lines · On Thu, 8 Jan 2009 10:11:08 +0530 Balbir Singh <[email protected]>…
Quoted 9 lines · KAMEZAWA Hiroyuki
Quoted 6 lines · On Thu, 8 Jan 2009 09:55:58 +0530 Balbir Singh <[email protected]>…
Quoted 4 lines · KAMEZAWA Hiroyuki

Hmm, Could you clarify following ?

- Usage of memory at insertsion and usage of memory at reclaim is different. So, this *sorted* order by RB-tree isn't the best order in general.

Quoted 7 lines
Quoted 4 lines

True, but we frequently update the tree at an interval of HZ/4. Updating at every page fault sounded like an overkill and building the entire tree at reclaim is an overkill too.

"sort" is not necessary.
If this feature is implemented as background daemon, just select the worst one at each iteration is enough.

Quoted 6 lines

OK, definitely an alternative worth considering, but the trade-off is lazy building (your suggestion), which involves actively seeing the usage of all cgroups (and if they are large, O(c), c is number of cgroups can be quite a bit) versus building the tree as and when the fault occurs and controlled by some interval.

I never think there will be "thousands" of memcg. O(c) is not so bad if it's on background.

But usual cost of adding res_counter_soft_limit_excess(&mem->res); is big... This maintainance cost of tree is always necessary even while there are no memory shortage.

BTW,
- mutex is bad. Can you use mutex while __GFP_WAIT is unset ?

- what happens when a big uncharge() occurs and no new charge() happens ? please add

   +		mem = mem_cgroup_get_largest_soft_limit_exceeding_node();
		if ( mem is still over soft limit )
			do reclaim....

at least.

-Kame

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#2 lore raw

Re: [RFC][PATCH 1/4] Memory controller soft limit documentation

In reply to Balbir Singh
Quoted 4 lines · Balbir Singh

-7. TODO
+7. Soft limits

Documentation/controllers/memory.txt renamed from Documentation/controllers/memory.txt~memcg-soft-limit-documentation +2 -0
+
+Soft limits allow for greater sharing of memory. The idea behind soft limits

Documentation/controllers/memory.txt cgroups created below it. · line 375

Quoted 10 lines
Documentation/controllers/memory.txt renamed from Documentation/controllers/memory.txt~memcg-soft-limit-documentation +10 -0
+is to allow control groups to use as much of the memory as needed, provided
+
+a. There is no memory contention
+b. They do not exceed their hard limit
+
+When the system detects memory contention (through do_try_to_free_pages(),
+while allocating), control groups are pushed back to their soft limits if
+possible. If the soft limit of each control group is very high, they are
+pushed back as much as possible to make sure that one control group does not
+starve the others.

Can you give an example here of how to implement the following setup:

- we have a high-priority latency-sensitive server job A and a bunch of low-priority batch jobs B, C and D

- each job *may* need up to 2GB of memory, but generally each tends to use <1GB of memory

- we want to run all four jobs on a 4GB machine

- we don't want A to ever have to wait for memory to be reclaimed (as it's serving latency-sensitive queries), so the kernel should be squashing B/C/D down *before* memory actually runs out.

Is this possible with the proposed hard/soft limit setup? Or do we need some additional support for keeping a pool of pre-reserved free memory available?

Paul

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#3 lore raw

Re: [RFC][PATCH 1/4] Memory controller soft limit documentation

Quoted 26 lines · Paul Menage
Quoted 4 lines · Balbir Singh

-7. TODO
+7. Soft limits

Documentation/controllers/memory.txt renamed from Documentation/controllers/memory.txt~memcg-soft-limit-documentation +2 -0
+
+Soft limits allow for greater sharing of memory. The idea behind soft limits

Documentation/controllers/memory.txt cgroups created below it. · line 375

Quoted 10 lines
Documentation/controllers/memory.txt renamed from Documentation/controllers/memory.txt~memcg-soft-limit-documentation +10 -0
+is to allow control groups to use as much of the memory as needed, provided
+
+a. There is no memory contention
+b. They do not exceed their hard limit
+
+When the system detects memory contention (through do_try_to_free_pages(),
+while allocating), control groups are pushed back to their soft limits if
+possible. If the soft limit of each control group is very high, they are
+pushed back as much as possible to make sure that one control group does not
+starve the others.

Can you give an example here of how to implement the following setup:

- we have a high-priority latency-sensitive server job A and a bunch of low-priority batch jobs B, C and D

- each job *may* need up to 2GB of memory, but generally each tends to use <1GB of memory

- we want to run all four jobs on a 4GB machine

Quoted 7 lines

- we don't want A to ever have to wait for memory to be reclaimed (as it's serving latency-sensitive queries), so the kernel should be squashing B/C/D down *before* memory actually runs out.

Is this possible with the proposed hard/soft limit setup? Or do we need some additional support for keeping a pool of pre-reserved free memory available?

This is a more complex scenario, It sounds like B/C and D should be hard limited to 2G or another value, depending on how much you want to pre-reserve for A (all B/C and D should be in the same cgroup). Then you want to use soft limits within the B/C/D cgroup. You don't want to hard limit A, but just setup a 2G soft limit for it.

The notion of prioritized jobs and reservation does not exist yet, but once we support soft limits and overcommit via soft limits, we could consider looking at what design aspects would help with it.

Signature · 7 lines
	Balbir

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>