Skip to contentSkip to conversation

Memory controller soft limit patches (v7)

PATCH In review Balbir Singh 3 participants 54 messages Mar 19, 2009 – Mar 25, 2009

Subsystems (7)
lore
Share & tools
lore
#0 lore raw

cover Memory controller soft limit patches (v7)

From: Balbir Singh <[email protected]>

New Feature: Soft limits for memory resource controller.

Changelog v7...v6
1. Added checks in reclaim path to make sure we don't infinitely loop
2. Refactored reclaim options into a new patch
3. Tested several scenarios, see tests below

Changelog v6...v5
1. If the number of reclaimed pages are zero, select the next mem cgroup for reclamation
2. Fixed a bug, where key was being updated after insertion into the tree
3. Fixed a build issue, when CONFIG_MEM_RES_CTLR is not enabled

Changelog v5...v4
1. Several changes to the reclaim logic, please see the patch 4 (reclaim on contention). I've experimented with several possibilities for reclaim and chose to come back to this due to the excellent behaviour seen while testing the patchset.
2. Reduced the overhead of soft limits on resource counters very significantly. Reaim benchmark now shows almost no drop in performance.

Changelog v4...v3
1. Adopted suggestions from Kamezawa to do a per-zone-per-node reclaim while doing soft limit reclaim. We don't record priorities while doing soft reclaim
2. Some of the overheads associated with soft limits (like calculating excess each time) is eliminated
3. The time_after(jiffies, 0) bug has been fixed
4. Tasks are throttled if the mem cgroup they belong to is being soft reclaimed and at the same time tasks are increasing the memory footprint and causing the mem cgroup to exceed its soft limit.

Changelog v3...v2
1. Implemented several review comments from Kosaki-San and Kamezawa-San Please see individual changelogs for changes

Changelog v2...v1
1. Soft limits now support hierarchies
2. Use spinlocks instead of mutexes for synchronization of the RB tree

Here is v7 of the new soft limit implementation. Soft limits is a new feature for the memory resource controller, something similar has existed in the group scheduler in the form of shares. The CPU controllers interpretation of shares is very different though.

Soft limits are the most useful feature to have for environments where the administrator wants to overcommit the system, such that only on memory contention do the limits become active. The current soft limits implementation provides a soft_limit_in_bytes interface for the memory controller and not for memory+swap controller. The implementation maintains an RB-Tree of groups that exceed their soft limit and starts reclaiming from the group that exceeds this limit by the maximum amount.

So far I have the best test results with this patchset. I've experimented with several approaches and methods. I might be a little delayed in responding, I might have intermittent access to the internet for the next few days.

TODOs

1. The current implementation maintains the delta from the soft limit and pushes back groups to their soft limits, a ratio of delta/soft_limit might be more useful

Tests
-----

I've run two memory intensive workloads with differing soft limits and seen that they are pushed back to their soft limit on contention. Their usage was their soft limit plus additional memory that they were able to grab on the system. Soft limit can take a while before we see the expected results.

The other tests I've run are
1. Deletion of groups while soft limit is in progress in the hierarchy
2. Setting the soft limit to zero and running other groups with non-zero soft limits.
3. Setting the soft limit to zero and testing if the mem cgroup is able to use available memory
4. Tested the patches with hierarchy enabled
5. Tested with swapoff -a, to make sure we don't go into an infinite loop

Please review, comment.

Series
------

memcg-soft-limits-documentation.patch
memcg-soft-limits-interface.patch
memcg-soft-limits-organize.patch
memcg-soft-limits-refactor-reclaim-bits
memcg-soft-limits-reclaim-on-contention.patch

Signature · 7 lines
	Balbir

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
#1 lore raw

PATCH 1/5 Memory controller soft limit documentation (v7)

Feature: Add documentation for soft limits

From: Balbir Singh <[email protected]>

---

 Documentation/cgroups/memory.txt |   31 ++++++++++++++++++++++++++++++-
 1 files changed, 30 insertions(+), 1 deletions(-)
Documentation/cgroups/memory.txt +30 -1
index a98a7fe..c5f73d9 100644
@@ -360,7 +360,36 @@ cgroups created below it.
 
 NOTE2: This feature can be enabled/disabled per subtree.
 
-7. TODO
+7. Soft limits
+
+Soft limits allow for greater sharing of memory. The idea behind soft limits
+is to allow control groups to use as much of the memory as needed, provided
+
+a. There is no memory contention
+b. They do not exceed their hard limit
+
+When the system detects memory contention or low memory control groups
+are pushed back to their soft limits. If the soft limit of each control
+group is very high, they are pushed back as much as possible to make
+sure that one control group does not starve the others of memory.
+
+7.1 Interface
+
+Soft limits can be setup by using the following commands (in this example we
+assume a soft limit of 256 megabytes)
+
+# echo 256M > memory.soft_limit_in_bytes
+
+If we want to change this to 1G, we can at any time use
+
+# echo 1G > memory.soft_limit_in_bytes
+
+NOTE1: Soft limits take effect over a long period of time, since they involve
+       reclaiming memory for balancing between memory cgroups
+NOTE2: It is recommended to set the soft limit always below the hard limit,
+       otherwise the hard limit will take precedence.
+
+8. TODO
 
 1. Add support for accounting huge pages (as a separate controller)
 2. Make per-cgroup scanner reclaim not-shared pages first
Signature · 7 lines
	Balbir

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
#2 lore raw

PATCH 2/5 Memory controller soft limit interface (v7)

In reply to Balbir Singh

Feature: Add soft limits interface to resource counters

From: Balbir Singh <[email protected]>

Changelog v2...v1
1. Add support for res_counter_check_soft_limit_locked. This is used by the hierarchy code.

Add an interface to allow get/set of soft limits. Soft limits for memory plus swap controller (memsw) is currently not supported. Resource counters have been enhanced to support soft limits and new type RES_SOFT_LIMIT has been added. Unlike hard limits, soft limits can be directly set and do not need any reclaim or checks before setting them to a newer value.

Kamezawa-San raised a question as to whether soft limit should belong to res_counter. Since all resources understand the basic concepts of hard and soft limits, it is justified to add soft limits here. Soft limits are a generic resource usage feature, even file system quotas support soft limits.

---

 include/linux/res_counter.h |   58 +++++++++++++++++++++++++++++++++++++++++++
 kernel/res_counter.c        |    3 ++
 mm/memcontrol.c             |   20 +++++++++++++++
 3 files changed, 81 insertions(+), 0 deletions(-)
include/linux/res_counter.h +58 -0
index 4c5bcf6..5c821fd 100644
@@ -35,6 +35,10 @@ struct res_counter {
 	 */
 	unsigned long long limit;
 	/*
+	 * the limit that usage can be exceed
+	 */
+	unsigned long long soft_limit;
+	/*
 	 * the number of unsuccessful attempts to consume the resource
 	 */
 	unsigned long long failcnt;
@@ -85,6 +89,7 @@ enum {
 	RES_MAX_USAGE,
 	RES_LIMIT,
 	RES_FAILCNT,
+	RES_SOFT_LIMIT,
 };
 
 /*
@@ -130,6 +135,36 @@ static inline bool res_counter_limit_check_locked(struct res_counter *cnt)
 	return false;
 }
 
+static inline bool res_counter_soft_limit_check_locked(struct res_counter *cnt)
+{
+	if (cnt->usage < cnt->soft_limit)
+		return true;
+
+	return false;
+}
+
+/**
+ * Get the difference between the usage and the soft limit
+ * @cnt: The counter
+ *
+ * Returns 0 if usage is less than or equal to soft limit
+ * The difference between usage and soft limit, otherwise.
+ */
+static inline unsigned long long
+res_counter_soft_limit_excess(struct res_counter *cnt)
+{
+	unsigned long long excess;
+	unsigned long flags;
+
+	spin_lock_irqsave(&cnt->lock, flags);
+	if (cnt->usage <= cnt->soft_limit)
+		excess = 0;
+	else
+		excess = cnt->usage - cnt->soft_limit;
+	spin_unlock_irqrestore(&cnt->lock, flags);
+	return excess;
+}
+
 /*
  * Helper function to detect if the cgroup is within it's limit or
  * not. It's currently called from cgroup_rss_prepare()
@@ -145,6 +180,17 @@ static inline bool res_counter_check_under_limit(struct res_counter *cnt)
 	return ret;
 }
 
+static inline bool res_counter_check_under_soft_limit(struct res_counter *cnt)
+{
+	bool ret;
+	unsigned long flags;
+
+	spin_lock_irqsave(&cnt->lock, flags);
+	ret = res_counter_soft_limit_check_locked(cnt);
+	spin_unlock_irqrestore(&cnt->lock, flags);
+	return ret;
+}
+
 static inline void res_counter_reset_max(struct res_counter *cnt)
 {
 	unsigned long flags;
@@ -178,4 +224,16 @@ static inline int res_counter_set_limit(struct res_counter *cnt,
 	return ret;
 }
 
+static inline int
+res_counter_set_soft_limit(struct res_counter *cnt,
+				unsigned long long soft_limit)
+{
+	unsigned long flags;
+
+	spin_lock_irqsave(&cnt->lock, flags);
+	cnt->soft_limit = soft_limit;
+	spin_unlock_irqrestore(&cnt->lock, flags);
+	return 0;
+}
+
 #endif
kernel/res_counter.c +3 -0
index bf8e753..4e6dafe 100644
@@ -19,6 +19,7 @@ void res_counter_init(struct res_counter *counter, struct res_counter *parent)
 {
 	spin_lock_init(&counter->lock);
 	counter->limit = (unsigned long long)LLONG_MAX;
+	counter->soft_limit = (unsigned long long)LLONG_MAX;
 	counter->parent = parent;
 }
 
@@ -101,6 +102,8 @@ res_counter_member(struct res_counter *counter, int member)
 		return &counter->limit;
 	case RES_FAILCNT:
 		return &counter->failcnt;
+	case RES_SOFT_LIMIT:
+		return &counter->soft_limit;
 	};
 
 	BUG();
mm/memcontrol.c +20 -0
index 5de6be9..70bc992 100644
@@ -2002,6 +2002,20 @@ static int mem_cgroup_write(struct cgroup *cont, struct cftype *cft,
 		else
 			ret = mem_cgroup_resize_memsw_limit(memcg, val);
 		break;
+	case RES_SOFT_LIMIT:
+		ret = res_counter_memparse_write_strategy(buffer, &val);
+		if (ret)
+			break;
+		/*
+		 * For memsw, soft limits are hard to implement in terms
+		 * of semantics, for now, we support soft limits for
+		 * control without swap
+		 */
+		if (type == _MEM)
+			ret = res_counter_set_soft_limit(&memcg->res, val);
+		else
+			ret = -EINVAL;
+		break;
 	default:
 		ret = -EINVAL; /* should be BUG() ? */
 		break;
@@ -2251,6 +2265,12 @@ static struct cftype mem_cgroup_files[] = {
 		.read_u64 = mem_cgroup_read,
 	},
 	{
+		.name = "soft_limit_in_bytes",
+		.private = MEMFILE_PRIVATE(_MEM, RES_SOFT_LIMIT),
+		.write_string = mem_cgroup_write,
+		.read_u64 = mem_cgroup_read,
+	},
+	{
 		.name = "failcnt",
 		.private = MEMFILE_PRIVATE(_MEM, RES_FAILCNT),
 		.trigger = mem_cgroup_reset,
Signature · 7 lines
	Balbir

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
#3 lore raw

PATCH 3/5 Memory controller soft limit organize cgroups (v7)

In reply to Balbir Singh

Feature: Organize cgroups over soft limit in a RB-Tree

From: Balbir Singh <[email protected]>

Changelog v7...v6
1. Refactor the check and update logic. The goal is to allow the check logic to be modular, so that it can be revisited in the future if something more appropriate is found to be useful.

Changelog v6...v5
1. Update the key before inserting into RB tree. Without the current change it could take an additional iteration to get the key correct.

Changelog v5...v4
1. res_counter_uncharge has an additional parameter to indicate if the counter was over its soft limit, before uncharge.

Changelog v4...v3
1. Optimizations to ensure we don't uncessarily get res_counter values
2. Fixed a bug in usage of time_after()

Changelog v3...v2
1. Add only the ancestor to the RB-Tree
2. Use css_tryget/css_put instead of mem_cgroup_get/mem_cgroup_put

Changelog v2...v1
1. Add support for hierarchies
2. The res_counter that is highest in the hierarchy is returned on soft limit being exceeded. Since we do hierarchical reclaim and add all groups exceeding their soft limits, this approach seems to work well in practice.

This patch introduces a RB-Tree for storing memory cgroups that are over their soft limit. The overall goal is to

1. Add a memory cgroup to the RB-Tree when the soft limit is exceeded. We are careful about updates, updates take place only after a particular time interval has passed
2. We remove the node from the RB-Tree when the usage goes below the soft limit

The next set of patches will exploit the RB-Tree to get the group that is over its soft limit by the largest amount and reclaim from it, when we face memory contention.

---

 include/linux/res_counter.h |    6 +-
 kernel/res_counter.c        |   18 +++++
 mm/memcontrol.c             |  149 ++++++++++++++++++++++++++++++++++++++-----
 3 files changed, 151 insertions(+), 22 deletions(-)
include/linux/res_counter.h +4 -2
index 5c821fd..5bbf8b1 100644
@@ -112,7 +112,8 @@ void res_counter_init(struct res_counter *counter, struct res_counter *parent);
 int __must_check res_counter_charge_locked(struct res_counter *counter,
 		unsigned long val);
 int __must_check res_counter_charge(struct res_counter *counter,
-		unsigned long val, struct res_counter **limit_fail_at);
+		unsigned long val, struct res_counter **limit_fail_at,
+		struct res_counter **soft_limit_at);
 
 /*
  * uncharge - tell that some portion of the resource is released
@@ -125,7 +126,8 @@ int __must_check res_counter_charge(struct res_counter *counter,
  */
 
 void res_counter_uncharge_locked(struct res_counter *counter, unsigned long val);
-void res_counter_uncharge(struct res_counter *counter, unsigned long val);
+void res_counter_uncharge(struct res_counter *counter, unsigned long val,
+				bool *was_soft_limit_excess);
 
 static inline bool res_counter_limit_check_locked(struct res_counter *cnt)
 {
kernel/res_counter.c +16 -2
index 4e6dafe..51ec438 100644
@@ -37,17 +37,27 @@ int res_counter_charge_locked(struct res_counter *counter, unsigned long val)
 }
 
 int res_counter_charge(struct res_counter *counter, unsigned long val,
-			struct res_counter **limit_fail_at)
+			struct res_counter **limit_fail_at,
+			struct res_counter **soft_limit_fail_at)
 {
 	int ret;
 	unsigned long flags;
 	struct res_counter *c, *u;
 
 	*limit_fail_at = NULL;
+	if (soft_limit_fail_at)
+		*soft_limit_fail_at = NULL;
 	local_irq_save(flags);
 	for (c = counter; c != NULL; c = c->parent) {
 		spin_lock(&c->lock);
 		ret = res_counter_charge_locked(c, val);
+		/*
+		 * With soft limits, we return the highest ancestor
+		 * that exceeds its soft limit
+		 */
+		if (soft_limit_fail_at &&
+			!res_counter_soft_limit_check_locked(c))
+			*soft_limit_fail_at = c;
 		spin_unlock(&c->lock);
 		if (ret < 0) {
 			*limit_fail_at = c;
@@ -75,7 +85,8 @@ void res_counter_uncharge_locked(struct res_counter *counter, unsigned long val)
 	counter->usage -= val;
 }
 
-void res_counter_uncharge(struct res_counter *counter, unsigned long val)
+void res_counter_uncharge(struct res_counter *counter, unsigned long val,
+				bool *was_soft_limit_excess)
 {
 	unsigned long flags;
 	struct res_counter *c;
@@ -83,6 +94,9 @@ void res_counter_uncharge(struct res_counter *counter, unsigned long val)
 	local_irq_save(flags);
 	for (c = counter; c != NULL; c = c->parent) {
 		spin_lock(&c->lock);
+		if (c == counter && was_soft_limit_excess)
+			*was_soft_limit_excess =
+				!res_counter_soft_limit_check_locked(c);
 		res_counter_uncharge_locked(c, val);
 		spin_unlock(&c->lock);
 	}
mm/memcontrol.c +131 -18

295 lines not on this page · raw message

Signature · 7 lines
	Balbir

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
#4 lore raw

Re: [PATCH 3/5] Memory controller soft limit organize cgroups (v7)

Quoted 9 lines · On Thu, 19 Mar 2009 22:27:35 +0530 Balbir Singh <[email protected]>…

Feature: Organize cgroups over soft limit in a RB-Tree

From: Balbir Singh <[email protected]>

Changelog v7...v6
1. Refactor the check and update logic. The goal is to allow the check logic to be modular, so that it can be revisited in the future if something more appropriate is found to be useful.

One of my motivation to this was "reducing if" in res_counter charege... But ..plz see comment.

Quoted 89 lines

Changelog v6...v5
1. Update the key before inserting into RB tree. Without the current change it could take an additional iteration to get the key correct.

Changelog v5...v4
1. res_counter_uncharge has an additional parameter to indicate if the counter was over its soft limit, before uncharge.

Changelog v4...v3
1. Optimizations to ensure we don't uncessarily get res_counter values
2. Fixed a bug in usage of time_after()

Changelog v3...v2
1. Add only the ancestor to the RB-Tree
2. Use css_tryget/css_put instead of mem_cgroup_get/mem_cgroup_put

Changelog v2...v1
1. Add support for hierarchies
2. The res_counter that is highest in the hierarchy is returned on soft limit being exceeded. Since we do hierarchical reclaim and add all groups exceeding their soft limits, this approach seems to work well in practice.

This patch introduces a RB-Tree for storing memory cgroups that are over their soft limit. The overall goal is to

1. Add a memory cgroup to the RB-Tree when the soft limit is exceeded. We are careful about updates, updates take place only after a particular time interval has passed
2. We remove the node from the RB-Tree when the usage goes below the soft limit

The next set of patches will exploit the RB-Tree to get the group that is over its soft limit by the largest amount and reclaim from it, when we face memory contention.

---

 include/linux/res_counter.h |    6 +-
 kernel/res_counter.c        |   18 +++++
 mm/memcontrol.c             |  149 ++++++++++++++++++++++++++++++++++++++-----
 3 files changed, 151 insertions(+), 22 deletions(-)
include/linux/res_counter.h +4 -2
index 5c821fd..5bbf8b1 100644
@@ -112,7 +112,8 @@ void res_counter_init(struct res_counter *counter, struct res_counter *parent);
 int __must_check res_counter_charge_locked(struct res_counter *counter,
 		unsigned long val);
 int __must_check res_counter_charge(struct res_counter *counter,
-		unsigned long val, struct res_counter **limit_fail_at);
+		unsigned long val, struct res_counter **limit_fail_at,
+		struct res_counter **soft_limit_at);
 
 /*
  * uncharge - tell that some portion of the resource is released
@@ -125,7 +126,8 @@ int __must_check res_counter_charge(struct res_counter *counter,
  */
 
 void res_counter_uncharge_locked(struct res_counter *counter, unsigned long val);
-void res_counter_uncharge(struct res_counter *counter, unsigned long val);
+void res_counter_uncharge(struct res_counter *counter, unsigned long val,
+				bool *was_soft_limit_excess);
 
 static inline bool res_counter_limit_check_locked(struct res_counter *cnt)
 {
kernel/res_counter.c +4 -1
index 4e6dafe..51ec438 100644
@@ -37,17 +37,27 @@ int res_counter_charge_locked(struct res_counter *counter, unsigned long val)
 }
 
 int res_counter_charge(struct res_counter *counter, unsigned long val,
-			struct res_counter **limit_fail_at)
+			struct res_counter **limit_fail_at,
+			struct res_counter **soft_limit_fail_at)
 {
 	int ret;
 	unsigned long flags;
 	struct res_counter *c, *u;
 
 	*limit_fail_at = NULL;
+	if (soft_limit_fail_at)
+		*soft_limit_fail_at = NULL;
 	local_irq_save(flags);

kernel/res_counter.c int res_counter_charge_locked(struct res_counter *counter, unsigned long val) · line 60

Quoted 10 lines
kernel/res_counter.c +7 -0
 	for (c = counter; c != NULL; c = c->parent) {
 		spin_lock(&c->lock);
 		ret = res_counter_charge_locked(c, val);
+		/*
+		 * With soft limits, we return the highest ancestor
+		 * that exceeds its soft limit
+		 */
+		if (soft_limit_fail_at &&
+			!res_counter_soft_limit_check_locked(c))
+			*soft_limit_fail_at = c;

Is this correct way to go ? In following situation,

     A/       softlimit=1G usage=1.2G
       B1/     sfotlimit=400M usage=1G
         C/
       B2/    softlimit=400M usage=200M

"A" will be victim and both of B1 and B2 will be reclaim target, right ?

and I wonder we don't need *softlimit_failed_at*... here.

<snip>

Quoted 4 lines
kernel/res_counter.c +4 -0
+static bool mem_cgroup_soft_limit_check(struct mem_cgroup *mem,
+					bool over_soft_limit)
+{
+	unsigned long next_update;

kernel/res_counter.c int res_counter_charge_locked(struct res_counter *counter, unsigned long val)

Quoted 10 lines
kernel/res_counter.c +10 -0
+
+	if (!over_soft_limit)
+		return false;
+
+	next_update = mem->last_tree_update + MEM_CGROUP_TREE_UPDATE_INTERVAL;
+	if (time_after(jiffies, next_update))
+		return true;
+
+	return false;
+}

If I write, this function will be

static bool mem_cgroup_soft_limit_check(struct mem_cgroup *mem, struct res_counter *failed_at)

{
	next_update = mem->last_tree_update + MEM_CGROUP_TREE_UPDATE_INTERVAL;
	if (!time_after(jiffies, next_update))
		return true;
	/* check softlimit */
        for (c = &mem->res; !c; c= c->parent) {
		if (!res_counter_check_under_soft_limit(c)) {
			failed_at =c;
		}
	}
	return false;
}
	/*
	 * Insert just the ancestor, we should trickle down to the correct
	 * cgroup for reclaim, since the other nodes will be below their
	 * soft limit
	 */
        if (mem_cgroup_soft_limit_check(mem, &soft_fail_res)) {
		mem_over_soft_limit =
			mem_cgroup_from_res_counter(soft_fail_res, res);
		mem_cgroup_update_tree(mem_over_soft_limit);
	}

Then, we really do softlimit check once in interval.

Thanks,
-Kame

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#5 lore raw

Re: [PATCH 3/5] Memory controller soft limit organize cgroups (v7)

Quoted 117 lines · KAMEZAWA Hiroyuki
Quoted 9 lines · On Thu, 19 Mar 2009 22:27:35 +0530 Balbir Singh <[email protected]>…

Feature: Organize cgroups over soft limit in a RB-Tree

From: Balbir Singh <[email protected]>

Changelog v7...v6
1. Refactor the check and update logic. The goal is to allow the check logic to be modular, so that it can be revisited in the future if something more appropriate is found to be useful.

One of my motivation to this was "reducing if" in res_counter charege... But ..plz see comment.

Quoted 89 lines

Changelog v6...v5
1. Update the key before inserting into RB tree. Without the current change it could take an additional iteration to get the key correct.

Changelog v5...v4
1. res_counter_uncharge has an additional parameter to indicate if the counter was over its soft limit, before uncharge.

Changelog v4...v3
1. Optimizations to ensure we don't uncessarily get res_counter values
2. Fixed a bug in usage of time_after()

Changelog v3...v2
1. Add only the ancestor to the RB-Tree
2. Use css_tryget/css_put instead of mem_cgroup_get/mem_cgroup_put

Changelog v2...v1
1. Add support for hierarchies
2. The res_counter that is highest in the hierarchy is returned on soft limit being exceeded. Since we do hierarchical reclaim and add all groups exceeding their soft limits, this approach seems to work well in practice.

This patch introduces a RB-Tree for storing memory cgroups that are over their soft limit. The overall goal is to

1. Add a memory cgroup to the RB-Tree when the soft limit is exceeded. We are careful about updates, updates take place only after a particular time interval has passed
2. We remove the node from the RB-Tree when the usage goes below the soft limit

The next set of patches will exploit the RB-Tree to get the group that is over its soft limit by the largest amount and reclaim from it, when we face memory contention.

---

 include/linux/res_counter.h |    6 +-
 kernel/res_counter.c        |   18 +++++
 mm/memcontrol.c             |  149 ++++++++++++++++++++++++++++++++++++++-----
 3 files changed, 151 insertions(+), 22 deletions(-)
include/linux/res_counter.h +4 -2
index 5c821fd..5bbf8b1 100644
@@ -112,7 +112,8 @@ void res_counter_init(struct res_counter *counter, struct res_counter *parent);
 int __must_check res_counter_charge_locked(struct res_counter *counter,
 		unsigned long val);
 int __must_check res_counter_charge(struct res_counter *counter,
-		unsigned long val, struct res_counter **limit_fail_at);
+		unsigned long val, struct res_counter **limit_fail_at,
+		struct res_counter **soft_limit_at);
 
 /*
  * uncharge - tell that some portion of the resource is released
@@ -125,7 +126,8 @@ int __must_check res_counter_charge(struct res_counter *counter,
  */
 
 void res_counter_uncharge_locked(struct res_counter *counter, unsigned long val);
-void res_counter_uncharge(struct res_counter *counter, unsigned long val);
+void res_counter_uncharge(struct res_counter *counter, unsigned long val,
+				bool *was_soft_limit_excess);
 
 static inline bool res_counter_limit_check_locked(struct res_counter *cnt)
 {
kernel/res_counter.c +4 -1
index 4e6dafe..51ec438 100644
@@ -37,17 +37,27 @@ int res_counter_charge_locked(struct res_counter *counter, unsigned long val)
 }
 
 int res_counter_charge(struct res_counter *counter, unsigned long val,
-			struct res_counter **limit_fail_at)
+			struct res_counter **limit_fail_at,
+			struct res_counter **soft_limit_fail_at)
 {
 	int ret;
 	unsigned long flags;
 	struct res_counter *c, *u;
 
 	*limit_fail_at = NULL;
+	if (soft_limit_fail_at)
+		*soft_limit_fail_at = NULL;
 	local_irq_save(flags);

kernel/res_counter.c int res_counter_charge_locked(struct res_counter *counter, unsigned long val) · line 60

Quoted 10 lines
kernel/res_counter.c +7 -0
 	for (c = counter; c != NULL; c = c->parent) {
 		spin_lock(&c->lock);
 		ret = res_counter_charge_locked(c, val);
+		/*
+		 * With soft limits, we return the highest ancestor
+		 * that exceeds its soft limit
+		 */
+		if (soft_limit_fail_at &&
+			!res_counter_soft_limit_check_locked(c))
+			*soft_limit_fail_at = c;

Is this correct way to go ? In following situation,

Quoted 7 lines
     A/       softlimit=1G usage=1.2G
       B1/     sfotlimit=400M usage=1G
         C/
       B2/    softlimit=400M usage=200M

"A" will be victim and both of B1 and B2 will be reclaim target, right ?

Yes, you remember we discussed adding the oldest ancestor in an older version. It was your suggestion to add the highest ancestor, have you changed your mind?

Quoted 2 lines

and I wonder we don't need *softlimit_failed_at*... here.

Not sure I get your point, could you please clarify this?

Quoted 46 lines

<snip>

Quoted 4 lines
kernel/res_counter.c +4 -0
+static bool mem_cgroup_soft_limit_check(struct mem_cgroup *mem,
+					bool over_soft_limit)
+{
+	unsigned long next_update;

kernel/res_counter.c

Quoted 10 lines
kernel/res_counter.c +10 -0
+
+	if (!over_soft_limit)
+		return false;
+
+	next_update = mem->last_tree_update + MEM_CGROUP_TREE_UPDATE_INTERVAL;
+	if (time_after(jiffies, next_update))
+		return true;
+
+	return false;
+}

If I write, this function will be

static bool mem_cgroup_soft_limit_check(struct mem_cgroup *mem, struct res_counter *failed_at)

{
	next_update = mem->last_tree_update + MEM_CGROUP_TREE_UPDATE_INTERVAL;
	if (!time_after(jiffies, next_update))
		return true;
	/* check softlimit */
        for (c = &mem->res; !c; c= c->parent) {
		if (!res_counter_check_under_soft_limit(c)) {
			failed_at =c;
		}
	}
	return false;
}
	/*
	 * Insert just the ancestor, we should trickle down to the correct
	 * cgroup for reclaim, since the other nodes will be below their
	 * soft limit
	 */
        if (mem_cgroup_soft_limit_check(mem, &soft_fail_res)) {
		mem_over_soft_limit =
			mem_cgroup_from_res_counter(soft_fail_res, res);
		mem_cgroup_update_tree(mem_over_soft_limit);
	}
Quoted 1 line

Then, we really do softlimit check once in interval.

OK, so the trade-off is - every once per interval,
I need to walk up res_counters all over again, hold all locks and check. Like I mentioned earlier, with the current approach I've reduced the overhead significantly for non-users. Earlier I was seeing a small loss in output with reaim, but since I changed res_counter_uncharge to track soft limits, that difference is negligible now.

The issue I see with this approach is that if soft-limits were not enabled, even then we would need to walk up the hierarchy and do tests, where as embedding it in res_counter_charge, one simple check tells us we don't have more to do.

Signature · 7 lines
	Balbir

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
#6 lore raw

Re: [PATCH 3/5] Memory controller soft limit organize cgroups (v7)

Quoted 16 lines · On Sun, 22 Mar 2009 19:51:05 +0530 Balbir Singh <[email protected]>…
Quoted 7 lines
        if (mem_cgroup_soft_limit_check(mem, &soft_fail_res)) {
		mem_over_soft_limit =
			mem_cgroup_from_res_counter(soft_fail_res, res);
		mem_cgroup_update_tree(mem_over_soft_limit);
	}

Then, we really do softlimit check once in interval.

OK, so the trade-off is - every once per interval,
I need to walk up res_counters all over again, hold all locks and check. Like I mentioned earlier, with the current approach I've reduced the overhead significantly for non-users. Earlier I was seeing a small loss in output with reaim, but since I changed res_counter_uncharge to track soft limits, that difference is negligible now.

Quoted 5 lines

The issue I see with this approach is that if soft-limits were not enabled, even then we would need to walk up the hierarchy and do tests, where as embedding it in res_counter_charge, one simple check tells us we don't have more to do.

Not at all.

just check softlimit is enabled or not in mem_cgroup_soft_limit_check() by some flag.

Thanks,
-Kame

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#7 lore raw

Re: [PATCH 3/5] Memory controller soft limit organize cgroups (v7)

Quoted 19 lines · KAMEZAWA Hiroyuki
Quoted 16 lines · On Sun, 22 Mar 2009 19:51:05 +0530 Balbir Singh <[email protected]>…
Quoted 7 lines
        if (mem_cgroup_soft_limit_check(mem, &soft_fail_res)) {
		mem_over_soft_limit =
			mem_cgroup_from_res_counter(soft_fail_res, res);
		mem_cgroup_update_tree(mem_over_soft_limit);
	}

Then, we really do softlimit check once in interval.

OK, so the trade-off is - every once per interval,
I need to walk up res_counters all over again, hold all locks and check. Like I mentioned earlier, with the current approach I've reduced the overhead significantly for non-users. Earlier I was seeing a small loss in output with reaim, but since I changed res_counter_uncharge to track soft limits, that difference is negligible now.

Quoted 9 lines
Quoted 5 lines

The issue I see with this approach is that if soft-limits were not enabled, even then we would need to walk up the hierarchy and do tests, where as embedding it in res_counter_charge, one simple check tells us we don't have more to do.

Not at all.

just check softlimit is enabled or not in mem_cgroup_soft_limit_check() by some flag.

So far, we don't use flags, the default soft limit is LONGLONG_MAX, if hierarchy is enabled, we need to check all the way up. The only way we check over limit is via a comparison. Are you suggesting we cache the value or save a special flag whenever the soft limit is set to anything other than LONGLONG_MAX? It is an indication that we are using soft limits, but we still need to see if we exceed it.

Why are we trying to over optimize this path? Like I mentioned earlier, the degradation is down to the order of noise. Knuth, re-learnt several times that "premature optimization is the root of all evil". If we find an issue with performance, we can definitely go down the road you are suggesting.

Signature · 7 lines
	Balbir

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
#8 lore raw

Re: [PATCH 3/5] Memory controller soft limit organize cgroups (v7)

Quoted 31 lines · On Mon, 23 Mar 2009 09:04:04 +0530 Balbir Singh <[email protected]>…
Quoted 19 lines · KAMEZAWA Hiroyuki
Quoted 16 lines · On Sun, 22 Mar 2009 19:51:05 +0530 Balbir Singh <[email protected]>…
Quoted 7 lines
        if (mem_cgroup_soft_limit_check(mem, &soft_fail_res)) {
		mem_over_soft_limit =
			mem_cgroup_from_res_counter(soft_fail_res, res);
		mem_cgroup_update_tree(mem_over_soft_limit);
	}

Then, we really do softlimit check once in interval.

OK, so the trade-off is - every once per interval,
I need to walk up res_counters all over again, hold all locks and check. Like I mentioned earlier, with the current approach I've reduced the overhead significantly for non-users. Earlier I was seeing a small loss in output with reaim, but since I changed res_counter_uncharge to track soft limits, that difference is negligible now.

Quoted 9 lines
Quoted 5 lines

The issue I see with this approach is that if soft-limits were not enabled, even then we would need to walk up the hierarchy and do tests, where as embedding it in res_counter_charge, one simple check tells us we don't have more to do.

Not at all.

just check softlimit is enabled or not in mem_cgroup_soft_limit_check() by some flag.

Quoted 7 lines

So far, we don't use flags, the default soft limit is LONGLONG_MAX, if hierarchy is enabled, we need to check all the way up. The only way we check over limit is via a comparison. Are you suggesting we cache the value or save a special flag whenever the soft limit is set to anything other than LONGLONG_MAX? It is an indication that we are using soft limits, but we still need to see if we exceed it.

Hmm ok, then, what we have to do here is
"children's softlimit should not be greater than parent's".
or
"if no softlimit, make last_tree_update to be enough big (jiffies + 1year)" This will reduce the check.

Quoted 6 lines

Why are we trying to over optimize this path? Like I mentioned earlier, the degradation is down to the order of noise. Knuth, re-learnt several times that "premature optimization is the root of all evil". If we find an issue with performance, we can definitely go down the road you are suggesting.

I just don't like "check always even if unnecessary"

Thanks,
-Kame

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#9 lore raw

Re: [PATCH 3/5] Memory controller soft limit organize cgroups (v7)

Quoted 42 lines · KAMEZAWA Hiroyuki
Quoted 31 lines · On Mon, 23 Mar 2009 09:04:04 +0530 Balbir Singh <[email protected]>…
Quoted 19 lines · KAMEZAWA Hiroyuki
Quoted 16 lines · On Sun, 22 Mar 2009 19:51:05 +0530 Balbir Singh <[email protected]>…
Quoted 7 lines
        if (mem_cgroup_soft_limit_check(mem, &soft_fail_res)) {
		mem_over_soft_limit =
			mem_cgroup_from_res_counter(soft_fail_res, res);
		mem_cgroup_update_tree(mem_over_soft_limit);
	}

Then, we really do softlimit check once in interval.

OK, so the trade-off is - every once per interval,
I need to walk up res_counters all over again, hold all locks and check. Like I mentioned earlier, with the current approach I've reduced the overhead significantly for non-users. Earlier I was seeing a small loss in output with reaim, but since I changed res_counter_uncharge to track soft limits, that difference is negligible now.

Quoted 9 lines
Quoted 5 lines

The issue I see with this approach is that if soft-limits were not enabled, even then we would need to walk up the hierarchy and do tests, where as embedding it in res_counter_charge, one simple check tells us we don't have more to do.

Not at all.

just check softlimit is enabled or not in mem_cgroup_soft_limit_check() by some flag.

Quoted 7 lines

So far, we don't use flags, the default soft limit is LONGLONG_MAX, if hierarchy is enabled, we need to check all the way up. The only way we check over limit is via a comparison. Are you suggesting we cache the value or save a special flag whenever the soft limit is set to anything other than LONGLONG_MAX? It is an indication that we are using soft limits, but we still need to see if we exceed it.

Quoted 6 lines

Hmm ok, then, what we have to do here is
"children's softlimit should not be greater than parent's".
or
"if no softlimit, make last_tree_update to be enough big (jiffies + 1year)" This will reduce the check.

No... That breaks hierarchy and changes limit behaviour. Today a hard limit can be greater than parent, if so we bottle-neck at the parent and catch it. I am not changing semantics.

Quoted 9 lines
Quoted 6 lines

Why are we trying to over optimize this path? Like I mentioned earlier, the degradation is down to the order of noise. Knuth, re-learnt several times that "premature optimization is the root of all evil". If we find an issue with performance, we can definitely go down the road you are suggesting.

I just don't like "check always even if unnecessary"

We do that even for hard limits today. The price (if any) is paid on enabling those features. My tests don't show the overhead. If we do see them in the future, we can revisit.

Signature · 7 lines
	Balbir

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
#10 lore raw

Re: [PATCH 3/5] Memory controller soft limit organize cgroups (v7)

Quoted 62 lines · On Mon, 23 Mar 2009 09:45:59 +0530 Balbir Singh <[email protected]>…
Quoted 42 lines · KAMEZAWA Hiroyuki
Quoted 31 lines · On Mon, 23 Mar 2009 09:04:04 +0530 Balbir Singh <[email protected]>…
Quoted 19 lines · KAMEZAWA Hiroyuki
Quoted 16 lines · On Sun, 22 Mar 2009 19:51:05 +0530 Balbir Singh <[email protected]>…
Quoted 7 lines
        if (mem_cgroup_soft_limit_check(mem, &soft_fail_res)) {
		mem_over_soft_limit =
			mem_cgroup_from_res_counter(soft_fail_res, res);
		mem_cgroup_update_tree(mem_over_soft_limit);
	}

Then, we really do softlimit check once in interval.

OK, so the trade-off is - every once per interval,
I need to walk up res_counters all over again, hold all locks and check. Like I mentioned earlier, with the current approach I've reduced the overhead significantly for non-users. Earlier I was seeing a small loss in output with reaim, but since I changed res_counter_uncharge to track soft limits, that difference is negligible now.

Quoted 9 lines
Quoted 5 lines

The issue I see with this approach is that if soft-limits were not enabled, even then we would need to walk up the hierarchy and do tests, where as embedding it in res_counter_charge, one simple check tells us we don't have more to do.

Not at all.

just check softlimit is enabled or not in mem_cgroup_soft_limit_check() by some flag.

Quoted 7 lines

So far, we don't use flags, the default soft limit is LONGLONG_MAX, if hierarchy is enabled, we need to check all the way up. The only way we check over limit is via a comparison. Are you suggesting we cache the value or save a special flag whenever the soft limit is set to anything other than LONGLONG_MAX? It is an indication that we are using soft limits, but we still need to see if we exceed it.

Quoted 6 lines

Hmm ok, then, what we have to do here is
"children's softlimit should not be greater than parent's".
or
"if no softlimit, make last_tree_update to be enough big (jiffies + 1year)" This will reduce the check.

No... That breaks hierarchy and changes limit behaviour. Today a hard limit can be greater than parent, if so we bottle-neck at the parent and catch it. I am not changing semantics.

Quoted 7 lines
Quoted 6 lines

Why are we trying to over optimize this path? Like I mentioned earlier, the degradation is down to the order of noise. Knuth, re-learnt several times that "premature optimization is the root of all evil". If we find an issue with performance, we can definitely go down the road you are suggesting.

Quoted 7 lines
Quoted 2 lines

I just don't like "check always even if unnecessary"

We do that even for hard limits today. The price (if any) is paid on enabling those features. My tests don't show the overhead. If we do see them in the future, we can revisit.

ok, plz don't expext Ack from me.

Thanks,
-Kame

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#11 lore raw

Re: [PATCH 3/5] Memory controller soft limit organize cgroups (v7)

Quoted 65 lines · KAMEZAWA Hiroyuki
Quoted 62 lines · On Mon, 23 Mar 2009 09:45:59 +0530 Balbir Singh <[email protected]>…
Quoted 42 lines · KAMEZAWA Hiroyuki
Quoted 31 lines · On Mon, 23 Mar 2009 09:04:04 +0530 Balbir Singh <[email protected]>…
Quoted 19 lines · KAMEZAWA Hiroyuki
Quoted 16 lines · On Sun, 22 Mar 2009 19:51:05 +0530 Balbir Singh <[email protected]>…
Quoted 7 lines
        if (mem_cgroup_soft_limit_check(mem, &soft_fail_res)) {
		mem_over_soft_limit =
			mem_cgroup_from_res_counter(soft_fail_res, res);
		mem_cgroup_update_tree(mem_over_soft_limit);
	}

Then, we really do softlimit check once in interval.

OK, so the trade-off is - every once per interval,
I need to walk up res_counters all over again, hold all locks and check. Like I mentioned earlier, with the current approach I've reduced the overhead significantly for non-users. Earlier I was seeing a small loss in output with reaim, but since I changed res_counter_uncharge to track soft limits, that difference is negligible now.

Quoted 9 lines
Quoted 5 lines

The issue I see with this approach is that if soft-limits were not enabled, even then we would need to walk up the hierarchy and do tests, where as embedding it in res_counter_charge, one simple check tells us we don't have more to do.

Not at all.

just check softlimit is enabled or not in mem_cgroup_soft_limit_check() by some flag.

Quoted 7 lines

So far, we don't use flags, the default soft limit is LONGLONG_MAX, if hierarchy is enabled, we need to check all the way up. The only way we check over limit is via a comparison. Are you suggesting we cache the value or save a special flag whenever the soft limit is set to anything other than LONGLONG_MAX? It is an indication that we are using soft limits, but we still need to see if we exceed it.

Quoted 6 lines

Hmm ok, then, what we have to do here is
"children's softlimit should not be greater than parent's".
or
"if no softlimit, make last_tree_update to be enough big (jiffies + 1year)" This will reduce the check.

No... That breaks hierarchy and changes limit behaviour. Today a hard limit can be greater than parent, if so we bottle-neck at the parent and catch it. I am not changing semantics.

Quoted 7 lines
Quoted 6 lines

Why are we trying to over optimize this path? Like I mentioned earlier, the degradation is down to the order of noise. Knuth, re-learnt several times that "premature optimization is the root of all evil". If we find an issue with performance, we can definitely go down the road you are suggesting.

Quoted 8 lines
Quoted 7 lines
Quoted 2 lines

I just don't like "check always even if unnecessary"

We do that even for hard limits today. The price (if any) is paid on enabling those features. My tests don't show the overhead. If we do see them in the future, we can revisit.

ok, plz don't expext Ack from me.

We can agree to disagree, as long as you don't NACK the patches without any testing, I don't see why we can't go ahead. The current design for soft limit is very much in line with our hierarchical hard limit design.

I don't see why you are harping about something that you might think is a problem and want to over-optimize even without tests. Fix something when you can see the problem, on my system I don't see it. I am willing to consider alternatives or moving away from the current coding style *iff* it needs to be redone for better performance.

What I am proposing is that we do iterative development, get the functionality right and then if needed tune for performance.

Signature · 7 lines
	Balbir

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
#12 lore raw

Re: [PATCH 3/5] Memory controller soft limit organize cgroups (v7)

Quoted 6 lines · On Mon, 23 Mar 2009 13:52:44 +0530 Balbir Singh <[email protected]>…

I don't see why you are harping about something that you might think is a problem and want to over-optimize even without tests. Fix something when you can see the problem, on my system I don't see it. I am willing to consider alternatives or moving away from the current coding style *iff* it needs to be redone for better performance.

It's usually true that "For optimize system, don't do anything unnecessary". And the patch increase size of res_counter_charge from 236bytes to 295bytes. on my compliler.

And this is called at every charge if the check is unnecessary. (i.e. the _real_ check itself is done once in a HZ/?)

Thanks
-Kame

Quoted 14 lines

What I am proposing is that we do iterative development, get the functionality right and then if needed tune for performance.

Signature · 7 lines
	Balbir

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#13 lore raw

Re: [PATCH 3/5] Memory controller soft limit organize cgroups (v7)

Quoted 9 lines · KAMEZAWA Hiroyuki
Quoted 6 lines · On Mon, 23 Mar 2009 13:52:44 +0530 Balbir Singh <[email protected]>…

I don't see why you are harping about something that you might think is a problem and want to over-optimize even without tests. Fix something when you can see the problem, on my system I don't see it. I am willing to consider alternatives or moving away from the current coding style *iff* it needs to be redone for better performance.

Quoted 4 lines

It's usually true that "For optimize system, don't do anything unnecessary". And the patch increase size of res_counter_charge from 236bytes to 295bytes. on my compliler.

New features do come with a cost, I expected to add 8 bytes not 59 bytes. Something is wrong, could you help me with offsets and what you see on your system.

Quoted 3 lines

And this is called at every charge if the check is unnecessary. (i.e. the _real_ check itself is done once in a HZ/?)

So your suggestions is

1. Add a flag to indicate if soft limits are enabled (new atomic field or field protected by a lock).
2. Every HZ/? walk up the entire tree hold all locks and check if soft limit is exceeded
3. Or, restrict the child not to have soft limit greater than parent and break design

Signature · 7 lines
	Balbir

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
#14 lore raw

Re: [PATCH 3/5] Memory controller soft limit organize cgroups (v7)

In reply to Balbir Singh
Quoted 99 lines · On Thu, 19 Mar 2009 22:27:35 +0530 Balbir Singh <[email protected]>…

Feature: Organize cgroups over soft limit in a RB-Tree

From: Balbir Singh <[email protected]>

Changelog v7...v6
1. Refactor the check and update logic. The goal is to allow the check logic to be modular, so that it can be revisited in the future if something more appropriate is found to be useful.

Changelog v6...v5
1. Update the key before inserting into RB tree. Without the current change it could take an additional iteration to get the key correct.

Changelog v5...v4
1. res_counter_uncharge has an additional parameter to indicate if the counter was over its soft limit, before uncharge.

Changelog v4...v3
1. Optimizations to ensure we don't uncessarily get res_counter values
2. Fixed a bug in usage of time_after()

Changelog v3...v2
1. Add only the ancestor to the RB-Tree
2. Use css_tryget/css_put instead of mem_cgroup_get/mem_cgroup_put

Changelog v2...v1
1. Add support for hierarchies
2. The res_counter that is highest in the hierarchy is returned on soft limit being exceeded. Since we do hierarchical reclaim and add all groups exceeding their soft limits, this approach seems to work well in practice.

This patch introduces a RB-Tree for storing memory cgroups that are over their soft limit. The overall goal is to

1. Add a memory cgroup to the RB-Tree when the soft limit is exceeded. We are careful about updates, updates take place only after a particular time interval has passed
2. We remove the node from the RB-Tree when the usage goes below the soft limit

The next set of patches will exploit the RB-Tree to get the group that is over its soft limit by the largest amount and reclaim from it, when we face memory contention.

---

 include/linux/res_counter.h |    6 +-
 kernel/res_counter.c        |   18 +++++
 mm/memcontrol.c             |  149 ++++++++++++++++++++++++++++++++++++++-----
 3 files changed, 151 insertions(+), 22 deletions(-)
include/linux/res_counter.h +4 -2
index 5c821fd..5bbf8b1 100644
@@ -112,7 +112,8 @@ void res_counter_init(struct res_counter *counter, struct res_counter *parent);
 int __must_check res_counter_charge_locked(struct res_counter *counter,
 		unsigned long val);
 int __must_check res_counter_charge(struct res_counter *counter,
-		unsigned long val, struct res_counter **limit_fail_at);
+		unsigned long val, struct res_counter **limit_fail_at,
+		struct res_counter **soft_limit_at);
 
 /*
  * uncharge - tell that some portion of the resource is released
@@ -125,7 +126,8 @@ int __must_check res_counter_charge(struct res_counter *counter,
  */
 
 void res_counter_uncharge_locked(struct res_counter *counter, unsigned long val);
-void res_counter_uncharge(struct res_counter *counter, unsigned long val);
+void res_counter_uncharge(struct res_counter *counter, unsigned long val,
+				bool *was_soft_limit_excess);
 
 static inline bool res_counter_limit_check_locked(struct res_counter *cnt)
 {
kernel/res_counter.c +4 -1
index 4e6dafe..51ec438 100644
@@ -37,17 +37,27 @@ int res_counter_charge_locked(struct res_counter *counter, unsigned long val)
 }
 
 int res_counter_charge(struct res_counter *counter, unsigned long val,
-			struct res_counter **limit_fail_at)
+			struct res_counter **limit_fail_at,
+			struct res_counter **soft_limit_fail_at)
 {
 	int ret;
 	unsigned long flags;
 	struct res_counter *c, *u;
 
 	*limit_fail_at = NULL;
+	if (soft_limit_fail_at)
+		*soft_limit_fail_at = NULL;
 	local_irq_save(flags);
 	for (c = counter; c != NULL; c = c->parent) {

kernel/res_counter.c int res_counter_charge_locked(struct res_counter *counter, unsigned long val) · line 61

Quoted 10 lines
kernel/res_counter.c +7 -0
 		spin_lock(&c->lock);
 		ret = res_counter_charge_locked(c, val);
+		/*
+		 * With soft limits, we return the highest ancestor
+		 * that exceeds its soft limit
+		 */
+		if (soft_limit_fail_at &&
+			!res_counter_soft_limit_check_locked(c))
+			*soft_limit_fail_at = c;
 		spin_unlock(&c->lock);

I'm not sure this works as intended or not. Could you clarify ? (see below)

    In following hierarchy,

         A/   soft_limit=1G, usage=1.2G.
           B  soft_limit=200M, usage=1G
           C  soft_limit=800M, usage=200M

This function returns only "A".
And memory will be reclaimed from B and C, at first.

Quoted 13 lines
kernel/res_counter.c +2 -1
 		if (ret < 0) {
 			*limit_fail_at = c;
@@ -75,7 +85,8 @@ void res_counter_uncharge_locked(struct res_counter *counter, unsigned long val)
 	counter->usage -= val;
 }
 
-void res_counter_uncharge(struct res_counter *counter, unsigned long val)
+void res_counter_uncharge(struct res_counter *counter, unsigned long val,
+				bool *was_soft_limit_excess)
 {
 	unsigned long flags;
 	struct res_counter *c;

kernel/res_counter.c void res_counter_uncharge(struct res_counter *counter, unsigned long val) · line 102

Quoted 9 lines
kernel/res_counter.c +3 -0
@@ -83,6 +94,9 @@ void res_counter_uncharge(struct res_counter *counter, unsigned long val)
 	local_irq_save(flags);
 	for (c = counter; c != NULL; c = c->parent) {
 		spin_lock(&c->lock);
+		if (c == counter && was_soft_limit_excess)
+			*was_soft_limit_excess =
+				!res_counter_soft_limit_check_locked(c);
 		res_counter_uncharge_locked(c, val);
 		spin_unlock(&c->lock);
 	}

Does this work as intended ?
Assume following hierarchy

A/ softlimit=1G usage=300M

     B/ softlimit=200M usage=300M.
     C/ softlimit=800M usage=0M

*was_soft_limit_excess will be false and no tree update, forever.

Hmm ?

Thanks,
-Kame

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#15 lore raw

Re: [PATCH 3/5] Memory controller soft limit organize cgroups (v7)

Quoted 115 lines · KAMEZAWA Hiroyuki
Quoted 99 lines · On Thu, 19 Mar 2009 22:27:35 +0530 Balbir Singh <[email protected]>…

Feature: Organize cgroups over soft limit in a RB-Tree

From: Balbir Singh <[email protected]>

Changelog v7...v6
1. Refactor the check and update logic. The goal is to allow the check logic to be modular, so that it can be revisited in the future if something more appropriate is found to be useful.

Changelog v6...v5
1. Update the key before inserting into RB tree. Without the current change it could take an additional iteration to get the key correct.

Changelog v5...v4
1. res_counter_uncharge has an additional parameter to indicate if the counter was over its soft limit, before uncharge.

Changelog v4...v3
1. Optimizations to ensure we don't uncessarily get res_counter values
2. Fixed a bug in usage of time_after()

Changelog v3...v2
1. Add only the ancestor to the RB-Tree
2. Use css_tryget/css_put instead of mem_cgroup_get/mem_cgroup_put

Changelog v2...v1
1. Add support for hierarchies
2. The res_counter that is highest in the hierarchy is returned on soft limit being exceeded. Since we do hierarchical reclaim and add all groups exceeding their soft limits, this approach seems to work well in practice.

This patch introduces a RB-Tree for storing memory cgroups that are over their soft limit. The overall goal is to

1. Add a memory cgroup to the RB-Tree when the soft limit is exceeded. We are careful about updates, updates take place only after a particular time interval has passed
2. We remove the node from the RB-Tree when the usage goes below the soft limit

The next set of patches will exploit the RB-Tree to get the group that is over its soft limit by the largest amount and reclaim from it, when we face memory contention.

---

 include/linux/res_counter.h |    6 +-
 kernel/res_counter.c        |   18 +++++
 mm/memcontrol.c             |  149 ++++++++++++++++++++++++++++++++++++++-----
 3 files changed, 151 insertions(+), 22 deletions(-)
include/linux/res_counter.h +4 -2
index 5c821fd..5bbf8b1 100644
@@ -112,7 +112,8 @@ void res_counter_init(struct res_counter *counter, struct res_counter *parent);
 int __must_check res_counter_charge_locked(struct res_counter *counter,
 		unsigned long val);
 int __must_check res_counter_charge(struct res_counter *counter,
-		unsigned long val, struct res_counter **limit_fail_at);
+		unsigned long val, struct res_counter **limit_fail_at,
+		struct res_counter **soft_limit_at);
 
 /*
  * uncharge - tell that some portion of the resource is released
@@ -125,7 +126,8 @@ int __must_check res_counter_charge(struct res_counter *counter,
  */
 
 void res_counter_uncharge_locked(struct res_counter *counter, unsigned long val);
-void res_counter_uncharge(struct res_counter *counter, unsigned long val);
+void res_counter_uncharge(struct res_counter *counter, unsigned long val,
+				bool *was_soft_limit_excess);
 
 static inline bool res_counter_limit_check_locked(struct res_counter *cnt)
 {
kernel/res_counter.c +4 -1
index 4e6dafe..51ec438 100644
@@ -37,17 +37,27 @@ int res_counter_charge_locked(struct res_counter *counter, unsigned long val)
 }
 
 int res_counter_charge(struct res_counter *counter, unsigned long val,
-			struct res_counter **limit_fail_at)
+			struct res_counter **limit_fail_at,
+			struct res_counter **soft_limit_fail_at)
 {
 	int ret;
 	unsigned long flags;
 	struct res_counter *c, *u;
 
 	*limit_fail_at = NULL;
+	if (soft_limit_fail_at)
+		*soft_limit_fail_at = NULL;
 	local_irq_save(flags);
 	for (c = counter; c != NULL; c = c->parent) {

kernel/res_counter.c int res_counter_charge_locked(struct res_counter *counter, unsigned long val) · line 61

Quoted 10 lines
kernel/res_counter.c +7 -0
 		spin_lock(&c->lock);
 		ret = res_counter_charge_locked(c, val);
+		/*
+		 * With soft limits, we return the highest ancestor
+		 * that exceeds its soft limit
+		 */
+		if (soft_limit_fail_at &&
+			!res_counter_soft_limit_check_locked(c))
+			*soft_limit_fail_at = c;
 		spin_unlock(&c->lock);

I'm not sure this works as intended or not. Could you clarify ? (see below)

Quoted 9 lines
    In following hierarchy,

         A/   soft_limit=1G, usage=1.2G.
           B  soft_limit=200M, usage=1G
           C  soft_limit=800M, usage=200M

This function returns only "A".
And memory will be reclaimed from B and C, at first.

Yes, A will be put on the RB-Tree, earlier we were putting both A and B, but you were opposed to it (I can't remember why). Memory will be reclaimed from A, B and C through hierarchical reclaim.

Quoted 27 lines
Quoted 15 lines
kernel/res_counter.c +2 -1
 		if (ret < 0) {
 			*limit_fail_at = c;
@@ -75,7 +85,8 @@ void res_counter_uncharge_locked(struct res_counter *counter, unsigned long val)
 	counter->usage -= val;
 }
 
-void res_counter_uncharge(struct res_counter *counter, unsigned long val)
+void res_counter_uncharge(struct res_counter *counter, unsigned long val,
+				bool *was_soft_limit_excess)
 {
 	unsigned long flags;
 	struct res_counter *c;

kernel/res_counter.c void res_counter_uncharge(struct res_counter *counter, unsigned long val) · line 102

Quoted 9 lines
kernel/res_counter.c +3 -0
@@ -83,6 +94,9 @@ void res_counter_uncharge(struct res_counter *counter, unsigned long val)
 	local_irq_save(flags);
 	for (c = counter; c != NULL; c = c->parent) {
 		spin_lock(&c->lock);
+		if (c == counter && was_soft_limit_excess)
+			*was_soft_limit_excess =
+				!res_counter_soft_limit_check_locked(c);
 		res_counter_uncharge_locked(c, val);
 		spin_unlock(&c->lock);
 	}

Does this work as intended ?
Assume following hierarchy

Quoted 6 lines
   A/  softlimit=1G usage=300M
     B/ softlimit=200M usage=300M.
     C/ softlimit=800M usage=0M

*was_soft_limit_excess will be false and no tree update, forever.

No.. was_soft_limit_excess checks the soft limit before uncharge to see if we were over soft limit, when a page gets uncharged from B, since B is over soft limit and on tree, we will update the tree. Why do you say that was_soft_limit_excess will return false?

Signature · 7 lines
	Balbir

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
#16 lore raw

Re: [PATCH 3/5] Memory controller soft limit organize cgroups (v7)

Quoted 29 lines · On Wed, 25 Mar 2009 10:59:47 +0530 Balbir Singh <[email protected]>…
Quoted 27 lines · KAMEZAWA Hiroyuki
Quoted 11 lines
kernel/res_counter.c +2 -1
@@ -75,7 +85,8 @@ void res_counter_uncharge_locked(struct res_counter *counter, unsigned long val)
 	counter->usage -= val;
 }
 
-void res_counter_uncharge(struct res_counter *counter, unsigned long val)
+void res_counter_uncharge(struct res_counter *counter, unsigned long val,
+				bool *was_soft_limit_excess)
 {
 	unsigned long flags;
 	struct res_counter *c;

kernel/res_counter.c void res_counter_uncharge(struct res_counter *counter, unsigned long val) · line 102

Quoted 9 lines
kernel/res_counter.c +3 -0
@@ -83,6 +94,9 @@ void res_counter_uncharge(struct res_counter *counter, unsigned long val)
 	local_irq_save(flags);
 	for (c = counter; c != NULL; c = c->parent) {
 		spin_lock(&c->lock);
+		if (c == counter && was_soft_limit_excess)
+			*was_soft_limit_excess =
+				!res_counter_soft_limit_check_locked(c);
 		res_counter_uncharge_locked(c, val);
 		spin_unlock(&c->lock);
 	}

Does this work as intended ?
Assume following hierarchy

   A/  softlimit=1G usage=300M
     B/ softlimit=200M usage=300M.
     C/ softlimit=800M usage=0M
Quoted 8 lines
Quoted 2 lines

*was_soft_limit_excess will be false and no tree update, forever.

No.. was_soft_limit_excess checks the soft limit before uncharge to see if we were over soft limit, when a page gets uncharged from B, since B is over soft limit and on tree, we will update the tree. Why do you say that was_soft_limit_excess will return false?

my eyes tend to be buggy. ok, change the question.

==
+void res_counter_uncharge(struct res_counter *counter, unsigned long val,
+ bool *was_soft_limit_excess)
{
unsigned long flags;
struct res_counter *c;
@@ -83,6 +94,9 @@ void res_counter_uncharge(struct res_counter *counter, unsigned long val)
local_irq_save(flags);
for (c = counter; c != NULL; c = c->parent) {
spin_lock(&c->lock);
+ if (c == counter && was_soft_limit_excess)
+ *was_soft_limit_excess =
+ !res_counter_soft_limit_check_locked(c);
res_counter_uncharge_locked(c, val);
spin_unlock(&c->lock);
}
==
Why just check "c == coutner" case is enough ?

Thanks,
-Kame

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#17 lore raw

Re: [PATCH 3/5] Memory controller soft limit organize cgroups (v7)

Quoted 35 lines · KAMEZAWA Hiroyuki
Quoted 32 lines · On Wed, 25 Mar 2009 10:59:47 +0530 Balbir Singh <[email protected]>…
Quoted 29 lines · KAMEZAWA Hiroyuki
Quoted 11 lines
kernel/res_counter.c +2 -1
@@ -75,7 +85,8 @@ void res_counter_uncharge_locked(struct res_counter *counter, unsigned long val)
 	counter->usage -= val;
 }
 
-void res_counter_uncharge(struct res_counter *counter, unsigned long val)
+void res_counter_uncharge(struct res_counter *counter, unsigned long val,
+				bool *was_soft_limit_excess)
 {
 	unsigned long flags;
 	struct res_counter *c;

kernel/res_counter.c void res_counter_uncharge(struct res_counter *counter, unsigned long val) · line 102

Quoted 9 lines
kernel/res_counter.c +3 -0
@@ -83,6 +94,9 @@ void res_counter_uncharge(struct res_counter *counter, unsigned long val)
 	local_irq_save(flags);
 	for (c = counter; c != NULL; c = c->parent) {
 		spin_lock(&c->lock);
+		if (c == counter && was_soft_limit_excess)
+			*was_soft_limit_excess =
+				!res_counter_soft_limit_check_locked(c);
 		res_counter_uncharge_locked(c, val);
 		spin_unlock(&c->lock);
 	}

Does this work as intended ?
Assume following hierarchy

   A/  softlimit=1G usage=300M
     B/ softlimit=200M usage=300M.
     C/ softlimit=800M usage=0M

*was_soft_limit_excess will be false and no tree update, forever.

Quoted 7 lines
Quoted 5 lines

No.. was_soft_limit_excess checks the soft limit before uncharge to see if we were over soft limit, when a page gets uncharged from B, since B is over soft limit and on tree, we will update the tree. Why do you say that was_soft_limit_excess will return false?

my eyes tend to be buggy. ok, change the question.

No problem, we've all been there :)

Quoted 16 lines

==

kernel/res_counter.c +5 -0
+void res_counter_uncharge(struct res_counter *counter, unsigned long val,
+				bool *was_soft_limit_excess)
 {
 	unsigned long flags;
 	struct res_counter *c;
@@ -83,6 +94,9 @@ void res_counter_uncharge(struct res_counter *counter, unsigned long val)
 	local_irq_save(flags);
 	for (c = counter; c != NULL; c = c->parent) {
 		spin_lock(&c->lock);
+		if (c == counter && was_soft_limit_excess)
+			*was_soft_limit_excess =
+				!res_counter_soft_limit_check_locked(c);
 		res_counter_uncharge_locked(c, val);
 		spin_unlock(&c->lock);
 	}
Quoted 3 lines

==
Why just check "c == coutner" case is enough ?

This is a very good question, I think this check might not be necessary and can also be potentially buggy.

Quoted 7 lines

Thanks,
-Kame

Signature · 7 lines
	Balbir

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
#18 lore raw

Re: [PATCH 3/5] Memory controller soft limit organize cgroups (v7)

Quoted 14 lines · On Wed, 25 Mar 2009 11:23:54 +0530 Balbir Singh <[email protected]>…
Quoted 14 lines

==

kernel/res_counter.c +5 -0
+void res_counter_uncharge(struct res_counter *counter, unsigned long val,
+				bool *was_soft_limit_excess)
 {
 	unsigned long flags;
 	struct res_counter *c;
@@ -83,6 +94,9 @@ void res_counter_uncharge(struct res_counter *counter, unsigned long val)
 	local_irq_save(flags);
 	for (c = counter; c != NULL; c = c->parent) {
 		spin_lock(&c->lock);
+		if (c == counter && was_soft_limit_excess)
+			*was_soft_limit_excess =
+				!res_counter_soft_limit_check_locked(c);
 		res_counter_uncharge_locked(c, val);
Quoted 9 lines
Quoted 5 lines
kernel/res_counter.c +0 -0
 		spin_unlock(&c->lock);
 	}

==
Why just check "c == coutner" case is enough ?

This is a very good question, I think this check might not be necessary and can also be potentially buggy.

I feel so, but can't think of good clean up.

Don't we remove this check at uncharge ? Anyway status can be updated at
- charge().
- reclaim

I'll seek this way in mine...

Thanks,
-Kame

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#19 lore raw

Re: [PATCH 3/5] Memory controller soft limit organize cgroups (v7)

Quoted 26 lines · KAMEZAWA Hiroyuki
Quoted 14 lines · On Wed, 25 Mar 2009 11:23:54 +0530 Balbir Singh <[email protected]>…
Quoted 14 lines

==

kernel/res_counter.c +5 -0
+void res_counter_uncharge(struct res_counter *counter, unsigned long val,
+				bool *was_soft_limit_excess)
 {
 	unsigned long flags;
 	struct res_counter *c;
@@ -83,6 +94,9 @@ void res_counter_uncharge(struct res_counter *counter, unsigned long val)
 	local_irq_save(flags);
 	for (c = counter; c != NULL; c = c->parent) {
 		spin_lock(&c->lock);
+		if (c == counter && was_soft_limit_excess)
+			*was_soft_limit_excess =
+				!res_counter_soft_limit_check_locked(c);
 		res_counter_uncharge_locked(c, val);
Quoted 9 lines
Quoted 5 lines
kernel/res_counter.c +0 -0
 		spin_unlock(&c->lock);
 	}

==
Why just check "c == coutner" case is enough ?

This is a very good question, I think this check might not be necessary and can also be potentially buggy.

Quoted 7 lines

I feel so, but can't think of good clean up.

Don't we remove this check at uncharge ? Anyway status can be updated at
- charge().
- reclaim

I'll seek this way in mine...

The check can be removed, let me do that and re-run the overhead tests.

Signature · 7 lines
	Balbir

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
#20 lore raw

Re: [PATCH 3/5] Memory controller soft limit organize cgroups (v7)

Quoted 30 lines · Balbir Singh
Quoted 28 lines · KAMEZAWA Hiroyuki
Quoted 14 lines · On Wed, 25 Mar 2009 11:23:54 +0530 Balbir Singh <[email protected]>…
Quoted 14 lines

==

(quoted diff) +5 -0
+void res_counter_uncharge(struct res_counter *counter, unsigned long val,
+				bool *was_soft_limit_excess)
 {
 	unsigned long flags;
 	struct res_counter *c;
@@ -83,6 +94,9 @@ void res_counter_uncharge(struct res_counter *counter, unsigned long val)
 	local_irq_save(flags);
 	for (c = counter; c != NULL; c = c->parent) {
 		spin_lock(&c->lock);
+		if (c == counter && was_soft_limit_excess)
+			*was_soft_limit_excess =
+				!res_counter_soft_limit_check_locked(c);
 		res_counter_uncharge_locked(c, val);
Quoted 9 lines
Quoted 5 lines
(quoted diff) +0 -0
 		spin_unlock(&c->lock);
 	}

==
Why just check "c == coutner" case is enough ?

This is a very good question, I think this check might not be necessary and can also be potentially buggy.

I feel so, but can't think of good clean up.

Quoted 8 lines
Quoted 5 lines

Don't we remove this check at uncharge ? Anyway status can be updated at
- charge().
- reclaim

I'll seek this way in mine...

The check can be removed, let me do that and re-run the overhead tests.

OK, no impact since I don't have soft limits or hierarchy enabled in the tests for overhead (I am testing overhead for non-users of the feature).

Signature · 7 lines
	Balbir

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
#21 lore raw

Re: [PATCH 3/5] Memory controller soft limit organize cgroups (v7)

In reply to Balbir Singh

mm/memcontrol.c static int __mem_cgroup_try_charge(struct mm_struct *mm, · line 1033

Quoted 4 lines · On Thu, 19 Mar 2009 22:27:35 +0530 Balbir Singh <[email protected]>…
mm/memcontrol.c +0 -0
@@ -938,16 +1031,17 @@ static int __mem_cgroup_try_charge(struct mm_struct *mm,
 		int ret;
 		bool noswap = false;
 

In logical, plz add
soft_fail_res = NULL, here.

mm/memcontrol.c static int __mem_cgroup_try_charge(struct mm_struct *mm, · line 1035

Quoted 3 lines
mm/memcontrol.c +2 -1
-		ret = res_counter_charge(&mem->res, PAGE_SIZE, &fail_res);
+		ret = res_counter_charge(&mem->res, PAGE_SIZE, &fail_res,
+						&soft_fail_res);

-Kame

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#22 lore raw

Re: [PATCH 3/5] Memory controller soft limit organize cgroups (v7)

Quoted 9 lines · KAMEZAWA Hiroyuki

mm/memcontrol.c static int __mem_cgroup_try_charge(struct mm_struct *mm, · line 1033

Quoted 4 lines · On Thu, 19 Mar 2009 22:27:35 +0530 Balbir Singh <[email protected]>…
mm/memcontrol.c +0 -0
@@ -938,16 +1031,17 @@ static int __mem_cgroup_try_charge(struct mm_struct *mm,
 		int ret;
 		bool noswap = false;
 

In logical, plz add
soft_fail_res = NULL, here.

As an optimization? OK, done!

Quoted 8 lines

mm/memcontrol.c static int __mem_cgroup_try_charge(struct mm_struct *mm, · line 1035

Quoted 4 lines
mm/memcontrol.c +2 -1
-		ret = res_counter_charge(&mem->res, PAGE_SIZE, &fail_res);
+		ret = res_counter_charge(&mem->res, PAGE_SIZE, &fail_res,
+						&soft_fail_res);

-Kame

Signature · 7 lines
	Balbir

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
#23 lore raw

Re: [PATCH 3/5] Memory controller soft limit organize cgroups (v7)

Quoted 12 lines · On Wed, 25 Mar 2009 10:48:17 +0530 Balbir Singh <[email protected]>…
Quoted 9 lines · KAMEZAWA Hiroyuki

mm/memcontrol.c static int __mem_cgroup_try_charge(struct mm_struct *mm, · line 1033

Quoted 4 lines · On Thu, 19 Mar 2009 22:27:35 +0530 Balbir Singh <[email protected]>…
mm/memcontrol.c +0 -0
@@ -938,16 +1031,17 @@ static int __mem_cgroup_try_charge(struct mm_struct *mm,
 		int ret;
 		bool noswap = false;
 

In logical, plz add
soft_fail_res = NULL, here.

Quoted 2 lines

As an optimization? OK, done!

Ah, sorry....I missed that pointer was automatically initilized to NULL in res_counter_charge().
plz ignore..

-Kame

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#24 lore raw

PATCH 4/5 Memory controller soft limit refactor reclaim flags (v7)

In reply to Balbir Singh

Impact: Refactor mem_cgroup_hierarchical_reclaim()

From: Balbir Singh <[email protected]>

This patch refactors the arguments passed to mem_cgroup_hierarchical_reclaim() into flags, so that new parameters don't have to be passed as we make the reclaim routine more flexible

---

 mm/memcontrol.c |   27 ++++++++++++++++++++-------
 1 files changed, 20 insertions(+), 7 deletions(-)
mm/memcontrol.c +20 -7
index f5b61b8..992aac8 100644
@@ -227,6 +227,14 @@ pcg_default_flags[NR_CHARGE_TYPE] = {
 #define MEMFILE_TYPE(val)	(((val) >> 16) & 0xffff)
 #define MEMFILE_ATTR(val)	((val) & 0xffff)
 
+/*
+ * Reclaim flags for mem_cgroup_hierarchical_reclaim
+ */
+#define MEM_CGROUP_RECLAIM_NOSWAP_BIT	0x0
+#define MEM_CGROUP_RECLAIM_NOSWAP	(1 << MEM_CGROUP_RECLAIM_NOSWAP_BIT)
+#define MEM_CGROUP_RECLAIM_SHRINK_BIT	0x1
+#define MEM_CGROUP_RECLAIM_SHRINK	(1 << MEM_CGROUP_RECLAIM_SHRINK_BIT)
+
 static void mem_cgroup_get(struct mem_cgroup *mem);
 static void mem_cgroup_put(struct mem_cgroup *mem);
 static struct mem_cgroup *parent_mem_cgroup(struct mem_cgroup *mem);
@@ -889,11 +897,14 @@ mem_cgroup_select_victim(struct mem_cgroup *root_mem)
  * If shrink==true, for avoiding to free too much, this returns immedieately.
  */
 static int mem_cgroup_hierarchical_reclaim(struct mem_cgroup *root_mem,
-				   gfp_t gfp_mask, bool noswap, bool shrink)
+						gfp_t gfp_mask,
+						unsigned long reclaim_options)
 {
 	struct mem_cgroup *victim;
 	int ret, total = 0;
 	int loop = 0;
+	bool noswap = reclaim_options & MEM_CGROUP_RECLAIM_NOSWAP;
+	bool shrink = reclaim_options & MEM_CGROUP_RECLAIM_SHRINK;
 
 	while (loop < 2) {
 		victim = mem_cgroup_select_victim(root_mem);
@@ -1029,7 +1040,7 @@ static int __mem_cgroup_try_charge(struct mm_struct *mm,
 
 	while (1) {
 		int ret;
-		bool noswap = false;
+		unsigned long flags = 0;
 
 		ret = res_counter_charge(&mem->res, PAGE_SIZE, &fail_res,
 						&soft_fail_res);
@@ -1042,7 +1053,7 @@ static int __mem_cgroup_try_charge(struct mm_struct *mm,
 				break;
 			/* mem+swap counter fails */
 			res_counter_uncharge(&mem->res, PAGE_SIZE, NULL);
-			noswap = true;
+			flags |= MEM_CGROUP_RECLAIM_NOSWAP;
 			mem_over_limit = mem_cgroup_from_res_counter(fail_res,
 									memsw);
 		} else
@@ -1054,7 +1065,7 @@ static int __mem_cgroup_try_charge(struct mm_struct *mm,
 			goto nomem;
 
 		ret = mem_cgroup_hierarchical_reclaim(mem_over_limit, gfp_mask,
-							noswap, false);
+							flags);
 		if (ret)
 			continue;
 
@@ -1766,7 +1777,7 @@ int mem_cgroup_shrink_usage(struct page *page,
 
 	do {
 		progress = mem_cgroup_hierarchical_reclaim(mem,
-					gfp_mask, true, false);
+					gfp_mask, MEM_CGROUP_RECLAIM_NOSWAP);
 		progress += mem_cgroup_check_under_limit(mem);
 	} while (!progress && --retry);
 
@@ -1821,7 +1832,7 @@ static int mem_cgroup_resize_limit(struct mem_cgroup *memcg,
 			break;
 
 		progress = mem_cgroup_hierarchical_reclaim(memcg, GFP_KERNEL,
-						   false, true);
+						   MEM_CGROUP_RECLAIM_SHRINK);
 		curusage = res_counter_read_u64(&memcg->res, RES_USAGE);
 		/* Usage is reduced ? */
   		if (curusage >= oldusage)
@@ -1869,7 +1880,9 @@ int mem_cgroup_resize_memsw_limit(struct mem_cgroup *memcg,
 		if (!ret)
 			break;
 
-		mem_cgroup_hierarchical_reclaim(memcg, GFP_KERNEL, true, true);
+		mem_cgroup_hierarchical_reclaim(memcg, GFP_KERNEL,
+						MEM_CGROUP_RECLAIM_NOSWAP |
+						MEM_CGROUP_RECLAIM_SHRINK);
 		curusage = res_counter_read_u64(&memcg->memsw, RES_USAGE);
 		/* Usage is reduced ? */
 		if (curusage >= oldusage)
Signature · 7 lines
	Balbir

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
#25 lore raw

Re: [PATCH 4/5] Memory controller soft limit refactor reclaim flags (v7)

Quoted 8 lines · On Thu, 19 Mar 2009 22:27:44 +0530 Balbir Singh <[email protected]>…

Impact: Refactor mem_cgroup_hierarchical_reclaim()

From: Balbir Singh <[email protected]>

This patch refactors the arguments passed to mem_cgroup_hierarchical_reclaim() into flags, so that new parameters don't have to be passed as we make the reclaim routine more flexible

seems nice :)

Thanks,
-Kame

Quoted 103 lines

---

 mm/memcontrol.c |   27 ++++++++++++++++++++-------
 1 files changed, 20 insertions(+), 7 deletions(-)
mm/memcontrol.c +20 -7
index f5b61b8..992aac8 100644
@@ -227,6 +227,14 @@ pcg_default_flags[NR_CHARGE_TYPE] = {
 #define MEMFILE_TYPE(val)	(((val) >> 16) & 0xffff)
 #define MEMFILE_ATTR(val)	((val) & 0xffff)
 
+/*
+ * Reclaim flags for mem_cgroup_hierarchical_reclaim
+ */
+#define MEM_CGROUP_RECLAIM_NOSWAP_BIT	0x0
+#define MEM_CGROUP_RECLAIM_NOSWAP	(1 << MEM_CGROUP_RECLAIM_NOSWAP_BIT)
+#define MEM_CGROUP_RECLAIM_SHRINK_BIT	0x1
+#define MEM_CGROUP_RECLAIM_SHRINK	(1 << MEM_CGROUP_RECLAIM_SHRINK_BIT)
+
 static void mem_cgroup_get(struct mem_cgroup *mem);
 static void mem_cgroup_put(struct mem_cgroup *mem);
 static struct mem_cgroup *parent_mem_cgroup(struct mem_cgroup *mem);
@@ -889,11 +897,14 @@ mem_cgroup_select_victim(struct mem_cgroup *root_mem)
  * If shrink==true, for avoiding to free too much, this returns immedieately.
  */
 static int mem_cgroup_hierarchical_reclaim(struct mem_cgroup *root_mem,
-				   gfp_t gfp_mask, bool noswap, bool shrink)
+						gfp_t gfp_mask,
+						unsigned long reclaim_options)
 {
 	struct mem_cgroup *victim;
 	int ret, total = 0;
 	int loop = 0;
+	bool noswap = reclaim_options & MEM_CGROUP_RECLAIM_NOSWAP;
+	bool shrink = reclaim_options & MEM_CGROUP_RECLAIM_SHRINK;
 
 	while (loop < 2) {
 		victim = mem_cgroup_select_victim(root_mem);
@@ -1029,7 +1040,7 @@ static int __mem_cgroup_try_charge(struct mm_struct *mm,
 
 	while (1) {
 		int ret;
-		bool noswap = false;
+		unsigned long flags = 0;
 
 		ret = res_counter_charge(&mem->res, PAGE_SIZE, &fail_res,
 						&soft_fail_res);
@@ -1042,7 +1053,7 @@ static int __mem_cgroup_try_charge(struct mm_struct *mm,
 				break;
 			/* mem+swap counter fails */
 			res_counter_uncharge(&mem->res, PAGE_SIZE, NULL);
-			noswap = true;
+			flags |= MEM_CGROUP_RECLAIM_NOSWAP;
 			mem_over_limit = mem_cgroup_from_res_counter(fail_res,
 									memsw);
 		} else
@@ -1054,7 +1065,7 @@ static int __mem_cgroup_try_charge(struct mm_struct *mm,
 			goto nomem;
 
 		ret = mem_cgroup_hierarchical_reclaim(mem_over_limit, gfp_mask,
-							noswap, false);
+							flags);
 		if (ret)
 			continue;
 
@@ -1766,7 +1777,7 @@ int mem_cgroup_shrink_usage(struct page *page,
 
 	do {
 		progress = mem_cgroup_hierarchical_reclaim(mem,
-					gfp_mask, true, false);
+					gfp_mask, MEM_CGROUP_RECLAIM_NOSWAP);
 		progress += mem_cgroup_check_under_limit(mem);
 	} while (!progress && --retry);
 
@@ -1821,7 +1832,7 @@ static int mem_cgroup_resize_limit(struct mem_cgroup *memcg,
 			break;
 
 		progress = mem_cgroup_hierarchical_reclaim(memcg, GFP_KERNEL,
-						   false, true);
+						   MEM_CGROUP_RECLAIM_SHRINK);
 		curusage = res_counter_read_u64(&memcg->res, RES_USAGE);
 		/* Usage is reduced ? */
   		if (curusage >= oldusage)
@@ -1869,7 +1880,9 @@ int mem_cgroup_resize_memsw_limit(struct mem_cgroup *memcg,
 		if (!ret)
 			break;
 
-		mem_cgroup_hierarchical_reclaim(memcg, GFP_KERNEL, true, true);
+		mem_cgroup_hierarchical_reclaim(memcg, GFP_KERNEL,
+						MEM_CGROUP_RECLAIM_NOSWAP |
+						MEM_CGROUP_RECLAIM_SHRINK);
 		curusage = res_counter_read_u64(&memcg->memsw, RES_USAGE);
 		/* Usage is reduced ? */
 		if (curusage >= oldusage)
Signature · 2 lines
-- 
	Balbir

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#26 lore raw

Re: [PATCH 4/5] Memory controller soft limit refactor reclaim flags (v7)

Quoted 5 lines · KAMEZAWA Hiroyuki
Quoted 2 lines · On Thu, 19 Mar 2009 22:27:44 +0530 Balbir Singh <[email protected]>…

Impact: Refactor mem_cgroup_hierarchical_reclaim()

Quoted 9 lines
Quoted 6 lines

From: Balbir Singh <[email protected]>

This patch refactors the arguments passed to mem_cgroup_hierarchical_reclaim() into flags, so that new parameters don't have to be passed as we make the reclaim routine more flexible

seems nice :)

Thanks!

Signature · 7 lines
	Balbir

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
#27 lore raw

PATCH 5/5 Memory controller soft limit reclaim on contention (v7)

In reply to Balbir Singh

Feature: Implement reclaim from groups over their soft limit

From: Balbir Singh <[email protected]>

Changelog v7...v6
1. Refactored out reclaim_options patch into a separate patch
2. Added additional checks for all swap off condition in mem_cgroup_hierarchical_reclaim()

Changelog v6...v5
1. Reclaim arguments to hierarchical reclaim have been merged into one parameter called reclaim_options.
2. Check if we failed to reclaim from one cgroup during soft reclaim, if so move on to the next one. This can be very useful if the zonelist passed to soft limit reclaim has no allocations from the selected memory cgroup
3. Coding style cleanups

Changelog v5...v4

1. Throttling is removed, earlier we throttled tasks over their soft limit
2. Reclaim has been moved back to __alloc_pages_internal, several experiments and tests showed that it was the best place to reclaim memory. kswapd has a different goal, that does not work with a single soft limit for the memory cgroup.
3. Soft limit reclaim is more targetted and the pages reclaim depend on the amount by which the soft limit is exceeded.

Changelog v4...v3
1. soft_reclaim is now called from balance_pgdat
2. soft_reclaim is aware of nodes and zones
3. A mem_cgroup will be throttled if it is undergoing soft limit reclaim and at the same time trying to allocate pages and exceed its soft limit.
4. A new mem_cgroup_shrink_zone() routine has been added to shrink zones particular to a mem cgroup.

Changelog v3...v2
1. Convert several arguments to hierarchical reclaim to flags, thereby consolidating them
2. The reclaim for soft limits is now triggered from kswapd
3. try_to_free_mem_cgroup_pages() now accepts an optional zonelist argument

Changelog v2...v1
1. Added support for hierarchical soft limits

This patch allows reclaim from memory cgroups on contention (via the direct reclaim path).

memory cgroup soft limit reclaim finds the group that exceeds its soft limit by the largest number of pages and reclaims pages from it and then reinserts the cgroup into its correct place in the rbtree.

Added additional checks to mem_cgroup_hierarchical_reclaim() to detect long loops in case all swap is turned off. The code has been refactored and the loop check (loop < 2) has been enhanced for soft limits. For soft limits, we try to do more targetted reclaim. Instead of bailing out after two loops, the routine now reclaims memory proportional to the size by which the soft limit is exceeded. The proportion has been empirically determined.

---

 include/linux/memcontrol.h |    8 ++
 include/linux/swap.h       |    1 
 mm/memcontrol.c            |  202 +++++++++++++++++++++++++++++++++++++++++---
 mm/page_alloc.c            |    9 ++
 mm/vmscan.c                |    5 +
 5 files changed, 206 insertions(+), 19 deletions(-)
include/linux/memcontrol.h +8 -0
index 18146c9..faeb358 100644
@@ -116,6 +116,8 @@ static inline bool mem_cgroup_disabled(void)
 }
 
 extern bool mem_cgroup_oom_called(struct task_struct *task);
+unsigned long mem_cgroup_soft_limit_reclaim(struct zonelist *zl,
+						gfp_t gfp_mask);
 
 #else /* CONFIG_CGROUP_MEM_RES_CTLR */
 struct mem_cgroup;
@@ -264,6 +266,12 @@ mem_cgroup_print_oom_info(struct mem_cgroup *memcg, struct task_struct *p)
 {
 }
 
+static inline
+unsigned long mem_cgroup_soft_limit_reclaim(struct zonelist *zl, gfp_t gfp_mask)
+{
+	return 0;
+}
+
 #endif /* CONFIG_CGROUP_MEM_CONT */
 
 #endif /* _LINUX_MEMCONTROL_H */
include/linux/swap.h +1 -0
index 989eb53..c128337 100644
@@ -215,6 +215,7 @@ static inline void lru_cache_add_active_file(struct page *page)
 extern unsigned long try_to_free_pages(struct zonelist *zonelist, int order,
 					gfp_t gfp_mask);
 extern unsigned long try_to_free_mem_cgroup_pages(struct mem_cgroup *mem,
+						  struct zonelist *zl,
 						  gfp_t gfp_mask, bool noswap,
 						  unsigned int swappiness);
 extern int __isolate_lru_page(struct page *page, int mode, int file);
mm/memcontrol.c +186 -16

311 lines not on this page · raw message

mm/page_alloc.c +8 -1
index f8fd1e2..5e1a6ca 100644
@@ -1598,7 +1598,14 @@ nofail_alloc:
 	reclaim_state.reclaimed_slab = 0;
 	p->reclaim_state = &reclaim_state;
 
-	did_some_progress = try_to_free_pages(zonelist, order, gfp_mask);
+	/*
+	 * Try to free up some pages from the memory controllers soft
+	 * limit queue.
+	 */
+	did_some_progress = mem_cgroup_soft_limit_reclaim(zonelist, gfp_mask);
+	if (order || !did_some_progress)
+		did_some_progress += try_to_free_pages(zonelist, order,
+							gfp_mask);
 
 	p->reclaim_state = NULL;
 	lockdep_clear_current_reclaim_state();
mm/vmscan.c +3 -2
index 5b560f9..0acd19d 100644
@@ -1708,6 +1708,7 @@ unsigned long try_to_free_pages(struct zonelist *zonelist, int order,
 #ifdef CONFIG_CGROUP_MEM_RES_CTLR
 
 unsigned long try_to_free_mem_cgroup_pages(struct mem_cgroup *mem_cont,
+					   struct zonelist *zonelist,
 					   gfp_t gfp_mask,
 					   bool noswap,
 					   unsigned int swappiness)
@@ -1721,14 +1722,14 @@ unsigned long try_to_free_mem_cgroup_pages(struct mem_cgroup *mem_cont,
 		.mem_cgroup = mem_cont,
 		.isolate_pages = mem_cgroup_isolate_pages,
 	};
-	struct zonelist *zonelist;
 
 	if (noswap)
 		sc.may_unmap = 0;
 
 	sc.gfp_mask = (gfp_mask & GFP_RECLAIM_MASK) |
 			(GFP_HIGHUSER_MOVABLE & ~GFP_RECLAIM_MASK);
-	zonelist = NODE_DATA(numa_node_id())->node_zonelists;
+	if (!zonelist)
+		zonelist = NODE_DATA(numa_node_id())->node_zonelists;
 	return do_try_to_free_pages(zonelist, &sc);
 }
 #endif
Signature · 7 lines
	Balbir

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
#28 lore raw

Re: [PATCH 5/5] Memory controller soft limit reclaim on contention (v7)

Quoted 10 lines · On Thu, 19 Mar 2009 22:27:52 +0530 Balbir Singh <[email protected]>…

Feature: Implement reclaim from groups over their soft limit

From: Balbir Singh <[email protected]>

Changelog v7...v6
1. Refactored out reclaim_options patch into a separate patch
2. Added additional checks for all swap off condition in mem_cgroup_hierarchical_reclaim()

mm/page_alloc.c nofail_alloc: · line 1608

Quoted 9 lines
mm/page_alloc.c +8 -1
-	did_some_progress = try_to_free_pages(zonelist, order, gfp_mask);
+	/*
+	 * Try to free up some pages from the memory controllers soft
+	 * limit queue.
+	 */
+	did_some_progress = mem_cgroup_soft_limit_reclaim(zonelist, gfp_mask);
+	if (order || !did_some_progress)
+		did_some_progress += try_to_free_pages(zonelist, order,
+							gfp_mask);

Anyway, my biggest concern is here, always.

        By this.
          if (order > 1), try_to_free_pages() is called twice.
        Hmm...how about

        if (!pages_reclaimed && !(gfp_mask & __GFP_NORETRY)) { # this is the first loop or noretry
               did_some_progress = mem_cgroup_soft_limit_reclaim(zonelist, gfp_mask);
               if (!did_some_progress)
                    did_some_progress = try_to_free_pages(zonelist, order, gfp_mask);
        }else
                    did_some_progress = try_to_free_pages(zonelist, order, gfp_mask);
        

        maybe a bit more concervative.


        And I wonder "nodemask" should be checked or not..
        softlimit reclaim doesn't seem to work well with nodemask...

Thanks,
-Kame

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#29 lore raw

Re: [PATCH 5/5] Memory controller soft limit reclaim on contention (v7)

Quoted 25 lines · KAMEZAWA Hiroyuki
Quoted 10 lines · On Thu, 19 Mar 2009 22:27:52 +0530 Balbir Singh <[email protected]>…

Feature: Implement reclaim from groups over their soft limit

From: Balbir Singh <[email protected]>

Changelog v7...v6
1. Refactored out reclaim_options patch into a separate patch
2. Added additional checks for all swap off condition in mem_cgroup_hierarchical_reclaim()

mm/page_alloc.c nofail_alloc: · line 1608

Quoted 9 lines
mm/page_alloc.c +8 -1
-	did_some_progress = try_to_free_pages(zonelist, order, gfp_mask);
+	/*
+	 * Try to free up some pages from the memory controllers soft
+	 * limit queue.
+	 */
+	did_some_progress = mem_cgroup_soft_limit_reclaim(zonelist, gfp_mask);
+	if (order || !did_some_progress)
+		did_some_progress += try_to_free_pages(zonelist, order,
+							gfp_mask);

Anyway, my biggest concern is here, always.

Quoted 2 lines

By this.
if (order > 1), try_to_free_pages() is called twice.

try_to_free_mem_cgroup_pages and try_to_free_pages() are called

Quoted 6 lines
        Hmm...how about



        if (!pages_reclaimed && !(gfp_mask & __GFP_NORETRY)) { # this is the first loop or noretry
               did_some_progress = mem_cgroup_soft_limit_reclaim(zonelist, gfp_mask);

OK, I see what you mean.. but the cost of the mem_cgroup_soft_limit_reclaim() is really a low overhead call, which will bail out very quickly if nothing is over their soft limit. Even if we retry, we do a simple check for soft-limit-reclaim, if there is really something to be reclaimed, we reclaim from there first.

Quoted 11 lines

if (!did_some_progress)
did_some_progress = try_to_free_pages(zonelist, order, gfp_mask);
}else
did_some_progress = try_to_free_pages(zonelist, order, gfp_mask);

maybe a bit more concervative.

And I wonder "nodemask" should be checked or not.. softlimit reclaim doesn't seem to work well with nodemask...

Doesn't the zonelist take care of nodemask?

Signature · 7 lines
	Balbir

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
#30 lore raw

Re: [PATCH 5/5] Memory controller soft limit reclaim on contention (v7)

Quoted 36 lines · On Sun, 22 Mar 2009 19:57:48 +0530 Balbir Singh <[email protected]>…
Quoted 25 lines · KAMEZAWA Hiroyuki
Quoted 10 lines · On Thu, 19 Mar 2009 22:27:52 +0530 Balbir Singh <[email protected]>…

Feature: Implement reclaim from groups over their soft limit

From: Balbir Singh <[email protected]>

Changelog v7...v6
1. Refactored out reclaim_options patch into a separate patch
2. Added additional checks for all swap off condition in mem_cgroup_hierarchical_reclaim()

mm/page_alloc.c nofail_alloc: · line 1608

Quoted 9 lines
mm/page_alloc.c +8 -1
-	did_some_progress = try_to_free_pages(zonelist, order, gfp_mask);
+	/*
+	 * Try to free up some pages from the memory controllers soft
+	 * limit queue.
+	 */
+	did_some_progress = mem_cgroup_soft_limit_reclaim(zonelist, gfp_mask);
+	if (order || !did_some_progress)
+		did_some_progress += try_to_free_pages(zonelist, order,
+							gfp_mask);

Anyway, my biggest concern is here, always.

Quoted 2 lines

By this.
if (order > 1), try_to_free_pages() is called twice.

try_to_free_mem_cgroup_pages and try_to_free_pages() are called

Quoted 4 lines

Hmm...how about

Quoted 6 lines
Quoted 2 lines
        if (!pages_reclaimed && !(gfp_mask & __GFP_NORETRY)) { # this is the first loop or noretry
               did_some_progress = mem_cgroup_soft_limit_reclaim(zonelist, gfp_mask);

OK, I see what you mean.. but the cost of the mem_cgroup_soft_limit_reclaim() is really a low overhead call, which will bail out very quickly if nothing is over their soft limit.

My point is "if something is over soft limit" case. Memory is reclaiemd twice. My above code tries to avoid call memory-reclaim twice.

Even if order > 0, mem_cgroup_try_to_free_pages() may be able to recover the situation. Maybe it's better to allow lumpty-reclaim even when !scanning_global_lru().

Quoted 4 lines

Even if we retry, we do a simple check for soft-limit-reclaim, if there is really something to be reclaimed, we reclaim from there first.

That means you reclaim memory twice ;)
AFAIK,
- fork() -> task_struct/stack
page table in x86 PAE mode
requires order-1 pages very frequently and this "call twice" approach will kill the application peformance very effectively.

Quoted 6 lines
Quoted 6 lines
               if (!did_some_progress)
                    did_some_progress = try_to_free_pages(zonelist, order, gfp_mask);
        }else
                    did_some_progress = try_to_free_pages(zonelist, order, gfp_mask);
Quoted 8 lines
Quoted 5 lines

maybe a bit more concervative.

And I wonder "nodemask" should be checked or not.. softlimit reclaim doesn't seem to work well with nodemask...

Doesn't the zonelist take care of nodemask?

Not sure, but I think, no check. hmm BUG in vmscan.c ?

Thanks,
-Kame

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#31 lore raw

Re: [PATCH 5/5] Memory controller soft limit reclaim on contention (v7)

Quoted 42 lines · KAMEZAWA Hiroyuki
Quoted 39 lines · On Sun, 22 Mar 2009 19:57:48 +0530 Balbir Singh <[email protected]>…
Quoted 25 lines · KAMEZAWA Hiroyuki
Quoted 10 lines · On Thu, 19 Mar 2009 22:27:52 +0530 Balbir Singh <[email protected]>…

Feature: Implement reclaim from groups over their soft limit

From: Balbir Singh <[email protected]>

Changelog v7...v6
1. Refactored out reclaim_options patch into a separate patch
2. Added additional checks for all swap off condition in mem_cgroup_hierarchical_reclaim()

mm/page_alloc.c nofail_alloc: · line 1608

Quoted 9 lines
mm/page_alloc.c +8 -1
-	did_some_progress = try_to_free_pages(zonelist, order, gfp_mask);
+	/*
+	 * Try to free up some pages from the memory controllers soft
+	 * limit queue.
+	 */
+	did_some_progress = mem_cgroup_soft_limit_reclaim(zonelist, gfp_mask);
+	if (order || !did_some_progress)
+		did_some_progress += try_to_free_pages(zonelist, order,
+							gfp_mask);

Anyway, my biggest concern is here, always.

Quoted 2 lines

By this.
if (order > 1), try_to_free_pages() is called twice.

try_to_free_mem_cgroup_pages and try_to_free_pages() are called

Quoted 6 lines
        Hmm...how about



        if (!pages_reclaimed && !(gfp_mask & __GFP_NORETRY)) { # this is the first loop or noretry
               did_some_progress = mem_cgroup_soft_limit_reclaim(zonelist, gfp_mask);
Quoted 7 lines
Quoted 3 lines

OK, I see what you mean.. but the cost of the mem_cgroup_soft_limit_reclaim() is really a low overhead call, which will bail out very quickly if nothing is over their soft limit.

My point is "if something is over soft limit" case. Memory is reclaiemd twice. My above code tries to avoid call memory-reclaim twice.

Twice if order > 0 or if soft limit reclaim fails or there is nothing to soft limit reclaim.

Quoted 4 lines

Even if order > 0, mem_cgroup_try_to_free_pages() may be able to recover the situation. Maybe it's better to allow lumpty-reclaim even when !scanning_global_lru().

if order > 0, we let the global reclaim handler reclaim (scan global LRU). I think the chance of success is higher through that path, having said that I have not experimented with trying to allow lumpy-reclaim from memory cgroup LRU's. I think that should be a separate effort from this one.

Quoted 11 lines
Quoted 5 lines

Even if we retry, we do a simple check for soft-limit-reclaim, if there is really something to be reclaimed, we reclaim from there first.

That means you reclaim memory twice ;)
AFAIK,
- fork() -> task_struct/stack
page table in x86 PAE mode
requires order-1 pages very frequently and this "call twice" approach will kill the application peformance very effectively.

Yes, it would if this was the only way to allocate pages. But look at reality, with kswapd running in the background, how frequently do you expect to hit the reclaim path. Could you clarify what you mean by order-1 (2^1), if so soft limit reclaim is not invoked and it should not hurt performance. What am I missing?

Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
               if (!did_some_progress)
                    did_some_progress = try_to_free_pages(zonelist, order, gfp_mask);
        }else
                    did_some_progress = try_to_free_pages(zonelist, order, gfp_mask);


        maybe a bit more concervative.
Quoted 8 lines
Quoted 5 lines
Quoted 2 lines

And I wonder "nodemask" should be checked or not.. softlimit reclaim doesn't seem to work well with nodemask...

Doesn't the zonelist take care of nodemask?

Not sure, but I think, no check. hmm BUG in vmscan.c ?

The zonelist is built using policy_zonelist, that handles nodemask as well. That should keep the zonelist and nodemask in sync.. no?

Signature · 7 lines
	Balbir

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
#32 lore raw

Re: [PATCH 5/5] Memory controller soft limit reclaim on contention (v7)

Quoted 11 lines · On Mon, 23 Mar 2009 09:42:53 +0530 Balbir Singh <[email protected]>…
Quoted 4 lines

Even if order > 0, mem_cgroup_try_to_free_pages() may be able to recover the situation. Maybe it's better to allow lumpty-reclaim even when !scanning_global_lru().

if order > 0, we let the global reclaim handler reclaim (scan global LRU). I think the chance of success is higher through that path, having said that I have not experimented with trying to allow lumpy-reclaim from memory cgroup LRU's. I think that should be a separate effort from this one.

But ignoring that will make the cost twice....

Quoted 12 lines
Quoted 11 lines
Quoted 5 lines

Even if we retry, we do a simple check for soft-limit-reclaim, if there is really something to be reclaimed, we reclaim from there first.

That means you reclaim memory twice ;)
AFAIK,
- fork() -> task_struct/stack
page table in x86 PAE mode
requires order-1 pages very frequently and this "call twice" approach will kill the application peformance very effectively.

Quoted 6 lines

Yes, it would if this was the only way to allocate pages. But look at reality, with kswapd running in the background, how frequently do you expect to hit the reclaim path. Could you clarify what you mean by order-1 (2^1), if so soft limit reclaim is not invoked and it should not hurt performance. What am I missing?

Hmm, maybe running hackbench under memory pressure will tell the answer. Anyway, plz get Ack from people for memory management.
Rik or Mel or Christoph or Nick or someone.

Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
Quoted 12 lines

if (!did_some_progress)
did_some_progress = try_to_free_pages(zonelist, order, gfp_mask);
}else
did_some_progress = try_to_free_pages(zonelist, order, gfp_mask);

maybe a bit more concervative.

And I wonder "nodemask" should be checked or not.. softlimit reclaim doesn't seem to work well with nodemask...

Quoted 9 lines
Quoted 5 lines
Quoted 2 lines

Doesn't the zonelist take care of nodemask?

Not sure, but I think, no check. hmm BUG in vmscan.c ?

The zonelist is built using policy_zonelist, that handles nodemask as well. That should keep the zonelist and nodemask in sync.. no?

I already sent a patch.

Thanks,
-Kame

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#33 lore raw

Re: [PATCH 5/5] Memory controller soft limit reclaim on contention (v7)

Quoted 8 lines · KAMEZAWA Hiroyuki
Quoted 5 lines · On Mon, 23 Mar 2009 09:42:53 +0530 Balbir Singh <[email protected]>…
Quoted 4 lines

Even if order > 0, mem_cgroup_try_to_free_pages() may be able to recover the situation. Maybe it's better to allow lumpty-reclaim even when !scanning_global_lru().

Quoted 9 lines
Quoted 6 lines

if order > 0, we let the global reclaim handler reclaim (scan global LRU). I think the chance of success is higher through that path, having said that I have not experimented with trying to allow lumpy-reclaim from memory cgroup LRU's. I think that should be a separate effort from this one.

But ignoring that will make the cost twice....

OK, lets fix it, but it as a separate effort and with data that shows us the same.

Quoted 18 lines
Quoted 12 lines
Quoted 11 lines
Quoted 5 lines

Even if we retry, we do a simple check for soft-limit-reclaim, if there is really something to be reclaimed, we reclaim from there first.

That means you reclaim memory twice ;)
AFAIK,
- fork() -> task_struct/stack
page table in x86 PAE mode
requires order-1 pages very frequently and this "call twice" approach will kill the application peformance very effectively.

Quoted 6 lines

Yes, it would if this was the only way to allocate pages. But look at reality, with kswapd running in the background, how frequently do you expect to hit the reclaim path. Could you clarify what you mean by order-1 (2^1), if so soft limit reclaim is not invoked and it should not hurt performance. What am I missing?

Quoted 4 lines

Hmm, maybe running hackbench under memory pressure will tell the answer. Anyway, plz get Ack from people for memory management.
Rik or Mel or Christoph or Nick or someone.

Rik is on the cc and is linux-mm. I hope they'll look at it.

Quoted 17 lines
Quoted 17 lines
Quoted 17 lines
Quoted 16 lines
Quoted 8 lines

if (!did_some_progress)
did_some_progress = try_to_free_pages(zonelist, order, gfp_mask);
}else
did_some_progress = try_to_free_pages(zonelist, order, gfp_mask);

Quoted 5 lines

maybe a bit more concervative.

And I wonder "nodemask" should be checked or not.. softlimit reclaim doesn't seem to work well with nodemask...

Doesn't the zonelist take care of nodemask?

Quoted 8 lines
Quoted 6 lines
Quoted 2 lines

Not sure, but I think, no check. hmm BUG in vmscan.c ?

The zonelist is built using policy_zonelist, that handles nodemask as well. That should keep the zonelist and nodemask in sync.. no?

I already sent a patch.

I've seen it, the basic assumption of the patch is that

policy_zonelist() and for_each_zone_zonelist_nodemask() where nodemask is derived from policy_nodemask() give different results.. correct?

Signature · 7 lines
	Balbir

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
#34 lore raw

Re: [PATCH 5/5] Memory controller soft limit reclaim on contention (v7)

Quoted 5 lines · On Mon, 23 Mar 2009 13:58:22 +0530 Balbir Singh <[email protected]>…

I've seen it, the basic assumption of the patch is that

policy_zonelist() and for_each_zone_zonelist_nodemask() where nodemask is derived from policy_nodemask() give different results.. correct?

Basic thinking is that there is alloc_pages_nodemask() but try_to_free_pages() ignores nodemask. Then, removing alloc_pages_nodemask() or taking care of nodemask in try_to_free_pages() is necessary.

How nodemask/zonelist is built is out of sight.

Thanks,
-Kame

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#35 lore raw
In reply to Balbir Singh
Quoted 85 lines · On Thu, 19 Mar 2009 22:27:13 +0530 Balbir Singh <[email protected]>…

From: Balbir Singh <[email protected]>

New Feature: Soft limits for memory resource controller.

Changelog v7...v6
1. Added checks in reclaim path to make sure we don't infinitely loop
2. Refactored reclaim options into a new patch
3. Tested several scenarios, see tests below

Changelog v6...v5
1. If the number of reclaimed pages are zero, select the next mem cgroup for reclamation
2. Fixed a bug, where key was being updated after insertion into the tree
3. Fixed a build issue, when CONFIG_MEM_RES_CTLR is not enabled

Changelog v5...v4
1. Several changes to the reclaim logic, please see the patch 4 (reclaim on contention). I've experimented with several possibilities for reclaim and chose to come back to this due to the excellent behaviour seen while testing the patchset.
2. Reduced the overhead of soft limits on resource counters very significantly. Reaim benchmark now shows almost no drop in performance.

Changelog v4...v3
1. Adopted suggestions from Kamezawa to do a per-zone-per-node reclaim while doing soft limit reclaim. We don't record priorities while doing soft reclaim
2. Some of the overheads associated with soft limits (like calculating excess each time) is eliminated
3. The time_after(jiffies, 0) bug has been fixed
4. Tasks are throttled if the mem cgroup they belong to is being soft reclaimed and at the same time tasks are increasing the memory footprint and causing the mem cgroup to exceed its soft limit.

Changelog v3...v2
1. Implemented several review comments from Kosaki-San and Kamezawa-San Please see individual changelogs for changes

Changelog v2...v1
1. Soft limits now support hierarchies
2. Use spinlocks instead of mutexes for synchronization of the RB tree

Here is v7 of the new soft limit implementation. Soft limits is a new feature for the memory resource controller, something similar has existed in the group scheduler in the form of shares. The CPU controllers interpretation of shares is very different though.

Soft limits are the most useful feature to have for environments where the administrator wants to overcommit the system, such that only on memory contention do the limits become active. The current soft limits implementation provides a soft_limit_in_bytes interface for the memory controller and not for memory+swap controller. The implementation maintains an RB-Tree of groups that exceed their soft limit and starts reclaiming from the group that exceeds this limit by the maximum amount.

So far I have the best test results with this patchset. I've experimented with several approaches and methods. I might be a little delayed in responding, I might have intermittent access to the internet for the next few days.

TODOs

1. The current implementation maintains the delta from the soft limit and pushes back groups to their soft limits, a ratio of delta/soft_limit might be more useful

Tests
-----

I've run two memory intensive workloads with differing soft limits and seen that they are pushed back to their soft limit on contention. Their usage was their soft limit plus additional memory that they were able to grab on the system. Soft limit can take a while before we see the expected results.

The other tests I've run are
1. Deletion of groups while soft limit is in progress in the hierarchy
2. Setting the soft limit to zero and running other groups with non-zero soft limits.
3. Setting the soft limit to zero and testing if the mem cgroup is able to use available memory
4. Tested the patches with hierarchy enabled
5. Tested with swapoff -a, to make sure we don't go into an infinite loop

Quoted 2 lines

Please review, comment.

please add text to explain the behaior, what happens in the following situation.

/group_A .....softlimit=100M usage=ANON=1G,FILE=1M /group_B .....softlimit=200M usage=ANON=1G,FILE=1M /group_C .....softlimit=300M
on swap-available/swap-less/swap-full system.

And Run run "dd" or "cp" of big files under group_C.

Thanks,
-Kame

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#36 lore raw
Quoted 94 lines · KAMEZAWA Hiroyuki
Quoted 85 lines · On Thu, 19 Mar 2009 22:27:13 +0530 Balbir Singh <[email protected]>…

From: Balbir Singh <[email protected]>

New Feature: Soft limits for memory resource controller.

Changelog v7...v6
1. Added checks in reclaim path to make sure we don't infinitely loop
2. Refactored reclaim options into a new patch
3. Tested several scenarios, see tests below

Changelog v6...v5
1. If the number of reclaimed pages are zero, select the next mem cgroup for reclamation
2. Fixed a bug, where key was being updated after insertion into the tree
3. Fixed a build issue, when CONFIG_MEM_RES_CTLR is not enabled

Changelog v5...v4
1. Several changes to the reclaim logic, please see the patch 4 (reclaim on contention). I've experimented with several possibilities for reclaim and chose to come back to this due to the excellent behaviour seen while testing the patchset.
2. Reduced the overhead of soft limits on resource counters very significantly. Reaim benchmark now shows almost no drop in performance.

Changelog v4...v3
1. Adopted suggestions from Kamezawa to do a per-zone-per-node reclaim while doing soft limit reclaim. We don't record priorities while doing soft reclaim
2. Some of the overheads associated with soft limits (like calculating excess each time) is eliminated
3. The time_after(jiffies, 0) bug has been fixed
4. Tasks are throttled if the mem cgroup they belong to is being soft reclaimed and at the same time tasks are increasing the memory footprint and causing the mem cgroup to exceed its soft limit.

Changelog v3...v2
1. Implemented several review comments from Kosaki-San and Kamezawa-San Please see individual changelogs for changes

Changelog v2...v1
1. Soft limits now support hierarchies
2. Use spinlocks instead of mutexes for synchronization of the RB tree

Here is v7 of the new soft limit implementation. Soft limits is a new feature for the memory resource controller, something similar has existed in the group scheduler in the form of shares. The CPU controllers interpretation of shares is very different though.

Soft limits are the most useful feature to have for environments where the administrator wants to overcommit the system, such that only on memory contention do the limits become active. The current soft limits implementation provides a soft_limit_in_bytes interface for the memory controller and not for memory+swap controller. The implementation maintains an RB-Tree of groups that exceed their soft limit and starts reclaiming from the group that exceeds this limit by the maximum amount.

So far I have the best test results with this patchset. I've experimented with several approaches and methods. I might be a little delayed in responding, I might have intermittent access to the internet for the next few days.

TODOs

1. The current implementation maintains the delta from the soft limit and pushes back groups to their soft limits, a ratio of delta/soft_limit might be more useful

Tests
-----

I've run two memory intensive workloads with differing soft limits and seen that they are pushed back to their soft limit on contention. Their usage was their soft limit plus additional memory that they were able to grab on the system. Soft limit can take a while before we see the expected results.

The other tests I've run are
1. Deletion of groups while soft limit is in progress in the hierarchy
2. Setting the soft limit to zero and running other groups with non-zero soft limits.
3. Setting the soft limit to zero and testing if the mem cgroup is able to use available memory
4. Tested the patches with hierarchy enabled
5. Tested with swapoff -a, to make sure we don't go into an infinite loop

Quoted 2 lines

Please review, comment.

please add text to explain the behaior, what happens in the following situation.

Quoted 6 lines
   /group_A .....softlimit=100M usage=ANON=1G,FILE=1M
   /group_B .....softlimit=200M usage=ANON=1G,FILE=1M
   /group_C .....softlimit=300M
   on swap-available/swap-less/swap-full system.

And Run run "dd" or "cp" of big files under group_C.

That depends on the memory on the system, on my system with 4G, things run just fine.

I tried the following

/group_A soft_limit=100M, needed memory=3200M (allocate and touch) /group_B soft_limit=200M, needed memory=3200M
/group_C soft_limit=300M, needed memory=1024M (dd in a while loop)

group_B and group_A had a difference of 200M in their allocations on average. group_C touched 800M as maximum usage in bytes and around 500M on the average.

With swap turned off

group_C was hit the most with a lot of reclaim taking place on it. group_A was OOM killed and immediately after group_B got all the memory it needed and completed successfully.

I have one large swap partition, so I could not test the partial-swap scenario.

Signature · 7 lines
	Balbir

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
#37 lore raw
Quoted 8 lines · On Mon, 23 Mar 2009 10:52:47 +0530 Balbir Singh <[email protected]>…
Quoted 8 lines

please add text to explain the behaior, what happens in the following situation.

   /group_A .....softlimit=100M usage=ANON=1G,FILE=1M
   /group_B .....softlimit=200M usage=ANON=1G,FILE=1M
   /group_C .....softlimit=300M
   on swap-available/swap-less/swap-full system.
Quoted 5 lines
Quoted 1 line

And Run run "dd" or "cp" of big files under group_C.

That depends on the memory on the system, on my system with 4G, things run just fine.

fine ?

Quoted 10 lines

I tried the following

/group_A soft_limit=100M, needed memory=3200M (allocate and touch) /group_B soft_limit=200M, needed memory=3200M
/group_C soft_limit=300M, needed memory=1024M (dd in a while loop)

group_B and group_A had a difference of 200M in their allocations on average. group_C touched 800M as maximum usage in bytes and around 500M on the average.

Quoted 5 lines

With swap turned off

group_C was hit the most with a lot of reclaim taking place on it. group_A was OOM killed and immediately after group_B got all the memory it needed and completed successfully.

Hmm ? OOM-Kill seems to happen without "dd" in group C...

Thanks,
-Kame

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#38 lore raw
In reply to Balbir Singh
Quoted 3 lines · On Mon, 23 Mar 2009 10:52:47 +0530 Balbir Singh <[email protected]>…

I have one large swap partition, so I could not test the partial-swap scenario.

plz go ahead as you like, Seems no landing point now and I'd like to see what I can, later. I'll send no ACK nor NACK, more.

But please get ack from someone resposible for glorbal memory reclaim. Especially for hooks in try_to_free_pages().

And please make it clear in documentation that
- Depends on the system but this may increase the usage of swap.
- Depends on the system but this may not work as the user expected as hard-limit.

Considering corner cases, this is a very complicated/usage-is-difficult feature.

-Kame

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#39 lore raw
Quoted 8 lines · On Mon, 23 Mar 2009 15:12:45 +0900 KAMEZAWA Hiroyuki…
Quoted 3 lines · On Mon, 23 Mar 2009 10:52:47 +0530 Balbir Singh <[email protected]>…

I have one large swap partition, so I could not test the partial-swap scenario.

plz go ahead as you like, Seems no landing point now and I'd like to see what I can, later. I'll send no ACK nor NACK, more.

But I dislike the whole concept, at all.

Thanks,
-Kame

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#40 lore raw
Quoted 5 lines
Quoted 2 lines · On Mon, 23 Mar 2009 15:12:45 +0900 KAMEZAWA Hiroyuki…

On Mon, 23 Mar 2009 10:52:47 +0530
Balbir Singh <[email protected]> wrote:

Quoted 7 lines
Quoted 6 lines
Quoted 3 lines

I have one large swap partition, so I could not test the partial-swap scenario.

plz go ahead as you like, Seems no landing point now and I'd like to see what I can, later. I'll send no ACK nor NACK, more.

But I dislike the whole concept, at all.

Kamezawa-san, This implementation is suck. but I think softlimit concept itself isn't suck.

So, I would suggested discuss this feature based on your
"memcg softlimit (Another one) v4" patch. I exept I can ack it after few spin.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#41 lore raw
Quoted 8 lines · KOSAKI Motohiro
Quoted 8 lines
Quoted 5 lines · On Mon, 23 Mar 2009 15:12:45 +0900 KAMEZAWA Hiroyuki…
Quoted 3 lines · On Mon, 23 Mar 2009 10:52:47 +0530 Balbir Singh <[email protected]>…

I have one large swap partition, so I could not test the partial-swap scenario.

Quoted 9 lines
Quoted 4 lines
Quoted 3 lines

plz go ahead as you like, Seems no landing point now and I'd like to see what I can, later. I'll send no ACK nor NACK, more.

But I dislike the whole concept, at all.

Kamezawa-san, This implementation is suck. but I think softlimit concept itself isn't suck.

Just because of the reclaim factor? Feel free to improve it iteratively. Like I said to Kamezawa, don't over optimize in the first iteration. Pre-mature optimization is the root of all evil.

Quoted 2 lines

So, I would suggested discuss this feature based on your
"memcg softlimit (Another one) v4" patch. I exept I can ack it after few spin.

Kame's implementation sucked quite badly, please see my posted test results. Basic, bare minimum functionality did not work.

Signature · 7 lines
	Balbir

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
#42 lore raw
Quoted 6 lines
Quoted 2 lines

Kamezawa-san, This implementation is suck. but I think softlimit concept itself isn't suck.

Just because of the reclaim factor? Feel free to improve it iteratively. Like I said to Kamezawa, don't over optimize in the first iteration. Pre-mature optimization is the root of all evil.

Agreed.
Then, I nacked premature optimization code everytime.

Quoted 5 lines
Quoted 2 lines

So, I would suggested discuss this feature based on your
"memcg softlimit (Another one) v4" patch. I exept I can ack it after few spin.

Kame's implementation sucked quite badly, please see my posted test results. Basic, bare minimum functionality did not work.

Yes. I see.
but I think it can be fixed. the basic design of the patch is sane IMHO.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#43 lore raw
Quoted 11 lines · KOSAKI Motohiro
Quoted 6 lines
Quoted 2 lines

Kamezawa-san, This implementation is suck. but I think softlimit concept itself isn't suck.

Just because of the reclaim factor? Feel free to improve it iteratively. Like I said to Kamezawa, don't over optimize in the first iteration. Pre-mature optimization is the root of all evil.

Agreed.
Then, I nacked premature optimization code everytime.

Quoted 9 lines
Quoted 5 lines
Quoted 2 lines

So, I would suggested discuss this feature based on your
"memcg softlimit (Another one) v4" patch. I exept I can ack it after few spin.

Kame's implementation sucked quite badly, please see my posted test results. Basic, bare minimum functionality did not work.

Yes. I see.
but I think it can be fixed. the basic design of the patch is sane IMHO.

I have the following major objections to design

1. The use of lists as a data-structure, it will not scale well.
2. Using zone watermarks to implement global soft limits

Signature · 7 lines
	Balbir

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
#44 lore raw
In reply to KAMEZAWA Hiroyuki
Quoted 5 lines · KAMEZAWA Hiroyuki
Quoted 2 lines · On Mon, 23 Mar 2009 15:12:45 +0900 KAMEZAWA Hiroyuki…

On Mon, 23 Mar 2009 10:52:47 +0530
Balbir Singh <[email protected]> wrote:

Quoted 8 lines
Quoted 6 lines
Quoted 3 lines

I have one large swap partition, so I could not test the partial-swap scenario.

plz go ahead as you like, Seems no landing point now and I'd like to see what I can, later. I'll send no ACK nor NACK, more.

But I dislike the whole concept, at all.

Kame, if you dislike it please don't enable memory.soft_limit_in_bytes. After having sent several revisions of your own patchset and helping me with review of several revisions, your sudden dislike comes as a surprise.

Please NOTE: I am not saying we'll never see any of the reclaim changes you are suggesting, all I am saying is lets do enough test to prove it is needed. Lets get the functionality right and then optimize if we have to.

Signature · 7 lines
	Balbir

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
#45 lore raw
Quoted 5 lines · On Mon, 23 Mar 2009 14:05:06 +0530 Balbir Singh <[email protected]>…

* KAMEZAWA Hiroyuki <[email protected]> [2009-03-23 15:17:03]:
Kame, if you dislike it please don't enable memory.soft_limit_in_bytes. After having sent several revisions of your own patchset and helping me with review of several revisions, your sudden dislike comes as a surprise.

I can't think
- we need hook in mem_cgroup_charge/uncharge.
- RB-tree is good.
- don't taking care of kswad is enough

and memcg should be independent from global memory reclaim AMAP.

Quoted 5 lines

Please NOTE: I am not saying we'll never see any of the reclaim changes you are suggesting, all I am saying is lets do enough test to prove it is needed. Lets get the functionality right and then optimize if we have to.

But this itself is problem for me.

When we added
- hierarchy
- swap handling
- etc...

Almost all bug reports are from Nishimura and Li Zefan, not from *us*.

Thanks,
-Kame

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#46 lore raw
Quoted 22 lines · KAMEZAWA Hiroyuki
Quoted 5 lines · On Mon, 23 Mar 2009 14:05:06 +0530 Balbir Singh <[email protected]>…

* KAMEZAWA Hiroyuki <[email protected]> [2009-03-23 15:17:03]:
Kame, if you dislike it please don't enable memory.soft_limit_in_bytes. After having sent several revisions of your own patchset and helping me with review of several revisions, your sudden dislike comes as a surprise.

I can't think
- we need hook in mem_cgroup_charge/uncharge.
- RB-tree is good.
- don't taking care of kswad is enough

and memcg should be independent from global memory reclaim AMAP.

Quoted 5 lines

Please NOTE: I am not saying we'll never see any of the reclaim changes you are suggesting, all I am saying is lets do enough test to prove it is needed. Lets get the functionality right and then optimize if we have to.

Quoted 9 lines

But this itself is problem for me.

When we added
- hierarchy
- swap handling
- etc...

Almost all bug reports are from Nishimura and Li Zefan, not from *us*.

As long as we fix them, I don't care who reports bugs. I've been testing the patches I have with various configurations, but not hard enough at times. The advantage of -mm is that we get enough testing through the contribution of folks like Li and Nishimura (which is very much appreciated). I am not asking for these patches to go into mainline, but for wider testing in -mm.

Signature · 7 lines
	Balbir

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
#47 lore raw
In reply to KAMEZAWA Hiroyuki
Quoted 8 lines · KAMEZAWA Hiroyuki
Quoted 3 lines · On Mon, 23 Mar 2009 10:52:47 +0530 Balbir Singh <[email protected]>…

I have one large swap partition, so I could not test the partial-swap scenario.

plz go ahead as you like, Seems no landing point now and I'd like to see what I can, later. I'll send no ACK nor NACK, more.

Quoted 7 lines

But please get ack from someone resposible for glorbal memory reclaim. Especially for hooks in try_to_free_pages().

And please make it clear in documentation that
- Depends on the system but this may increase the usage of swap.
- Depends on the system but this may not work as the user expected as hard-limit.

The documentation mentions that soft limits take a long time before coming into affect. The use of the word "soft" over "hard" and the usage of this terminology in resource management clearly implies what you say in point (2).

Quoted 5 lines

Considering corner cases, this is a very complicated/usage-is-difficult feature.

-Kame

Signature · 7 lines
	Balbir

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
#48 lore raw
In reply to Balbir Singh
Quoted 5 lines · On Thu, 19 Mar 2009 22:27:13 +0530 Balbir Singh <[email protected]>…

From: Balbir Singh <[email protected]>

New Feature: Soft limits for memory resource controller.

Hmm, did you test against _slow_ workload ?

I tried this. My system with 2cpus. 1.6GB of memory.

create 2 cgroups A and B. Assume allocate 1G of ANON in B. and run kernel-make in group A. (%make -j 4)

0. echo 3 > /proc/sys/drop_caches.
1. set A's softlimit to 300M.
2. alloc memory 1GB in group B.
3. run %make -j 4 in group A.
=> 250MB of memory are swapeed-out from B.

0. echo 3 > /proc/sys/drop_caches.
1. set A's softlimit to 10GB (never reach this.)
2. alloc memory 1GB in group B.
3. run %make -j 4 in group A.
=> 350MB of memory are swapeed-out from B.

Is this enough to your purpose ? 250MB of swap-out is too much.... (the number can be changed depends on enviroment.)

Thanks,
-Kame

Quoted 86 lines

Changelog v7...v6
1. Added checks in reclaim path to make sure we don't infinitely loop
2. Refactored reclaim options into a new patch
3. Tested several scenarios, see tests below

Changelog v6...v5
1. If the number of reclaimed pages are zero, select the next mem cgroup for reclamation
2. Fixed a bug, where key was being updated after insertion into the tree
3. Fixed a build issue, when CONFIG_MEM_RES_CTLR is not enabled

Changelog v5...v4
1. Several changes to the reclaim logic, please see the patch 4 (reclaim on contention). I've experimented with several possibilities for reclaim and chose to come back to this due to the excellent behaviour seen while testing the patchset.
2. Reduced the overhead of soft limits on resource counters very significantly. Reaim benchmark now shows almost no drop in performance.

Changelog v4...v3
1. Adopted suggestions from Kamezawa to do a per-zone-per-node reclaim while doing soft limit reclaim. We don't record priorities while doing soft reclaim
2. Some of the overheads associated with soft limits (like calculating excess each time) is eliminated
3. The time_after(jiffies, 0) bug has been fixed
4. Tasks are throttled if the mem cgroup they belong to is being soft reclaimed and at the same time tasks are increasing the memory footprint and causing the mem cgroup to exceed its soft limit.

Changelog v3...v2
1. Implemented several review comments from Kosaki-San and Kamezawa-San Please see individual changelogs for changes

Changelog v2...v1
1. Soft limits now support hierarchies
2. Use spinlocks instead of mutexes for synchronization of the RB tree

Here is v7 of the new soft limit implementation. Soft limits is a new feature for the memory resource controller, something similar has existed in the group scheduler in the form of shares. The CPU controllers interpretation of shares is very different though.

Soft limits are the most useful feature to have for environments where the administrator wants to overcommit the system, such that only on memory contention do the limits become active. The current soft limits implementation provides a soft_limit_in_bytes interface for the memory controller and not for memory+swap controller. The implementation maintains an RB-Tree of groups that exceed their soft limit and starts reclaiming from the group that exceeds this limit by the maximum amount.

So far I have the best test results with this patchset. I've experimented with several approaches and methods. I might be a little delayed in responding, I might have intermittent access to the internet for the next few days.

TODOs

1. The current implementation maintains the delta from the soft limit and pushes back groups to their soft limits, a ratio of delta/soft_limit might be more useful

Tests
-----

I've run two memory intensive workloads with differing soft limits and seen that they are pushed back to their soft limit on contention. Their usage was their soft limit plus additional memory that they were able to grab on the system. Soft limit can take a while before we see the expected results.

The other tests I've run are
1. Deletion of groups while soft limit is in progress in the hierarchy
2. Setting the soft limit to zero and running other groups with non-zero soft limits.
3. Setting the soft limit to zero and testing if the mem cgroup is able to use available memory
4. Tested the patches with hierarchy enabled
5. Tested with swapoff -a, to make sure we don't go into an infinite loop

Please review, comment.

Series
------

Quoted 10 lines

memcg-soft-limits-documentation.patch
memcg-soft-limits-interface.patch
memcg-soft-limits-organize.patch
memcg-soft-limits-refactor-reclaim-bits
memcg-soft-limits-reclaim-on-contention.patch

Signature · 1 line
	Balbir

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#49 lore raw
In reply to Balbir Singh
Quoted 76 lines · Balbir Singh

From: Balbir Singh <[email protected]>

New Feature: Soft limits for memory resource controller.

Changelog v7...v6
1. Added checks in reclaim path to make sure we don't infinitely loop
2. Refactored reclaim options into a new patch
3. Tested several scenarios, see tests below

Changelog v6...v5
1. If the number of reclaimed pages are zero, select the next mem cgroup for reclamation
2. Fixed a bug, where key was being updated after insertion into the tree
3. Fixed a build issue, when CONFIG_MEM_RES_CTLR is not enabled

Changelog v5...v4
1. Several changes to the reclaim logic, please see the patch 4 (reclaim on contention). I've experimented with several possibilities for reclaim and chose to come back to this due to the excellent behaviour seen while testing the patchset.
2. Reduced the overhead of soft limits on resource counters very significantly. Reaim benchmark now shows almost no drop in performance.

Changelog v4...v3
1. Adopted suggestions from Kamezawa to do a per-zone-per-node reclaim while doing soft limit reclaim. We don't record priorities while doing soft reclaim
2. Some of the overheads associated with soft limits (like calculating excess each time) is eliminated
3. The time_after(jiffies, 0) bug has been fixed
4. Tasks are throttled if the mem cgroup they belong to is being soft reclaimed and at the same time tasks are increasing the memory footprint and causing the mem cgroup to exceed its soft limit.

Changelog v3...v2
1. Implemented several review comments from Kosaki-San and Kamezawa-San Please see individual changelogs for changes

Changelog v2...v1
1. Soft limits now support hierarchies
2. Use spinlocks instead of mutexes for synchronization of the RB tree

Here is v7 of the new soft limit implementation. Soft limits is a new feature for the memory resource controller, something similar has existed in the group scheduler in the form of shares. The CPU controllers interpretation of shares is very different though.

Soft limits are the most useful feature to have for environments where the administrator wants to overcommit the system, such that only on memory contention do the limits become active. The current soft limits implementation provides a soft_limit_in_bytes interface for the memory controller and not for memory+swap controller. The implementation maintains an RB-Tree of groups that exceed their soft limit and starts reclaiming from the group that exceeds this limit by the maximum amount.

So far I have the best test results with this patchset. I've experimented with several approaches and methods. I might be a little delayed in responding, I might have intermittent access to the internet for the next few days.

TODOs

1. The current implementation maintains the delta from the soft limit and pushes back groups to their soft limits, a ratio of delta/soft_limit might be more useful

Tests
-----

I've run two memory intensive workloads with differing soft limits and seen that they are pushed back to their soft limit on contention. Their usage was their soft limit plus additional memory that they were able to grab on the system. Soft limit can take a while before we see the expected results.

Quoted 9 lines

The other tests I've run are
1. Deletion of groups while soft limit is in progress in the hierarchy
2. Setting the soft limit to zero and running other groups with non-zero soft limits.
3. Setting the soft limit to zero and testing if the mem cgroup is able to use available memory
4. Tested the patches with hierarchy enabled
5. Tested with swapoff -a, to make sure we don't go into an infinite loop

I've run lmbench with the soft limit patches and the results show no major overhead, there are some outliers and unexpected results.

The outliers are at context-switch 16p/64K, in communicating latencies and some unexpected results where the softlimit changes help improve performance (I consider these to be in the range of noise).

L M B E N C H 2 . 0 S U M M A R Y
------------------------------------

Basic system parameters
----------------------------------------------------
Host OS Description Mhz

--------- ------------- ----------------------- ----
nosoftlim Linux 2.6.29- x86_64-linux-gnu 2131
softlimit Linux 2.6.29- x86_64-linux-gnu 2131

Processor, Processes - times in microseconds - smaller is better
----------------------------------------------------------------
Host OS Mhz null null open selct sig sig fork exec sh
call I/O stat clos TCP inst hndl proc proc proc
--------- ------------- ---- ---- ---- ---- ---- ----- ---- ---- ---- ---- ----
nosoftlim Linux 2.6.29- 2131 0.67 1.33 29.9 36.8 6.484 1.12 12.1 508. 1708 6281 softlimit Linux 2.6.29- 2131 0.66 1.31 29.8 36.8 6.486 1.11 12.3 483. 1697 6241

Context switching - times in microseconds - smaller is better
-------------------------------------------------------------
Host OS 2p/0K 2p/16K 2p/64K 8p/16K 8p/64K 16p/16K 16p/64K
ctxsw ctxsw ctxsw ctxsw ctxsw ctxsw ctxsw
--------- ------------- ----- ------ ------ ------ ------ --------------
nosoftlim Linux 2.6.29- 2.190 9.2300 3.1900 9.7400 10.8 7.93000 4.36000
softlimit Linux 2.6.29- 0.970 4.8200 3.1300 8.8900 10.3 8.82000 10.7

*Local* Communication latencies in microseconds - smaller is better
-------------------------------------------------------------------
Host OS 2p/0K Pipe AF UDP RPC/ TCP RPC/ TCP
ctxsw UNIX UDP TCP conn
--------- ------------- ----- ----- ---- ----- ----- ----- ----- ----

nosoftlim Linux 2.6.29- 2.190  22.0 58.5  53.3  68.7  61.7  64.9 210.
softlimit Linux 2.6.29- 0.970  20.3 55.3  54.0  53.8  79.7  64.5 211.

File & VM system latencies in microseconds - smaller is better
--------------------------------------------------------------

Host                 OS   0K File      10K File      Mmap    Prot Page    
                        Create Delete Create Delete  Latency Fault Fault 
--------- ------------- ------ ------ ------ ------  ------- ----- ----- 
nosoftlim Linux 2.6.29-   51.6   48.6  153.6   87.4    20.2K 7.00000
softlimit Linux 2.6.29-   51.6   48.2  137.8   83.9    20.2K 6.00000

*Local* Communication bandwidths in MB/s - bigger is better
-----------------------------------------------------------
Host OS Pipe AF TCP File Mmap Bcopy Bcopy Mem
Mem
UNIX reread reread (libc) (hand) read write
--------- ------------- ---- ---- ---- ------ ------ ------ ------ ---- ----- nosoftlim Linux 2.6.29- 1367 778. 803. 2058.5 4659.4 1303.9 1303.5 4664 1422. softlimit Linux 2.6.29- 1314 823. 812. 2061.3 4659.9 1290.2 1280.9 4662 1422.

Memory latencies in nanoseconds - smaller is better
(WARNING - may not be correct, check graphs)
---------------------------------------------------

Host                 OS   Mhz  L1 $   L2 $    Main mem    Guesses
--------- -------------  ---- ----- ------    --------    -------
nosoftlim Linux 2.6.29-  2131 1.875 6.5990   76.8
softlimit Linux 2.6.29-  2131 1.875 6.5980   76.8

Earlier, I ran reaim and saw no regression there as well.

Signature · 7 lines
	Balbir

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
#50 lore raw
Quoted 7 lines · On Tue, 24 Mar 2009 23:04:14 +0530 Balbir Singh <[email protected]>…

I've run lmbench with the soft limit patches and the results show no major overhead, there are some outliers and unexpected results.

The outliers are at context-switch 16p/64K, in communicating latencies and some unexpected results where the softlimit changes help improve performance (I consider these to be in the range of noise).

ok, seems no regressions. but what is the softlimit value ? I think there result is of course souftlimit=0 case value...right ?

-Kame

Quoted 57 lines

L M B E N C H 2 . 0 S U M M A R Y
------------------------------------

Basic system parameters
----------------------------------------------------
Host OS Description Mhz

--------- ------------- ----------------------- ----
nosoftlim Linux 2.6.29- x86_64-linux-gnu 2131
softlimit Linux 2.6.29- x86_64-linux-gnu 2131

Processor, Processes - times in microseconds - smaller is better
----------------------------------------------------------------
Host OS Mhz null null open selct sig sig fork exec sh
call I/O stat clos TCP inst hndl proc proc proc
--------- ------------- ---- ---- ---- ---- ---- ----- ---- ---- ---- ---- ----
nosoftlim Linux 2.6.29- 2131 0.67 1.33 29.9 36.8 6.484 1.12 12.1 508. 1708 6281 softlimit Linux 2.6.29- 2131 0.66 1.31 29.8 36.8 6.486 1.11 12.3 483. 1697 6241

Context switching - times in microseconds - smaller is better
-------------------------------------------------------------
Host OS 2p/0K 2p/16K 2p/64K 8p/16K 8p/64K 16p/16K 16p/64K
ctxsw ctxsw ctxsw ctxsw ctxsw ctxsw ctxsw
--------- ------------- ----- ------ ------ ------ ------ --------------
nosoftlim Linux 2.6.29- 2.190 9.2300 3.1900 9.7400 10.8 7.93000 4.36000
softlimit Linux 2.6.29- 0.970 4.8200 3.1300 8.8900 10.3 8.82000 10.7

*Local* Communication latencies in microseconds - smaller is better
-------------------------------------------------------------------
Host OS 2p/0K Pipe AF UDP RPC/ TCP RPC/ TCP
ctxsw UNIX UDP TCP conn
--------- ------------- ----- ----- ---- ----- ----- ----- ----- ----

nosoftlim Linux 2.6.29- 2.190  22.0 58.5  53.3  68.7  61.7  64.9 210.
softlimit Linux 2.6.29- 0.970  20.3 55.3  54.0  53.8  79.7  64.5 211.

File & VM system latencies in microseconds - smaller is better
--------------------------------------------------------------

Host                 OS   0K File      10K File      Mmap    Prot Page    
                        Create Delete Create Delete  Latency Fault Fault 
--------- ------------- ------ ------ ------ ------  ------- ----- ----- 
nosoftlim Linux 2.6.29-   51.6   48.6  153.6   87.4    20.2K 7.00000
softlimit Linux 2.6.29-   51.6   48.2  137.8   83.9    20.2K 6.00000

*Local* Communication bandwidths in MB/s - bigger is better
-----------------------------------------------------------
Host OS Pipe AF TCP File Mmap Bcopy Bcopy Mem
Mem
UNIX reread reread (libc) (hand) read write
--------- ------------- ---- ---- ---- ------ ------ ------ ------ ---- ----- nosoftlim Linux 2.6.29- 1367 778. 803. 2058.5 4659.4 1303.9 1303.5 4664 1422. softlimit Linux 2.6.29- 1314 823. 812. 2061.3 4659.9 1290.2 1280.9 4662 1422.

Memory latencies in nanoseconds - smaller is better
(WARNING - may not be correct, check graphs)
---------------------------------------------------

Quoted 10 lines
Host                 OS   Mhz  L1 $   L2 $    Main mem    Guesses
--------- -------------  ---- ----- ------    --------    -------
nosoftlim Linux 2.6.29-  2131 1.875 6.5990   76.8
softlimit Linux 2.6.29-  2131 1.875 6.5980   76.8

Earlier, I ran reaim and saw no regression there as well.

Signature · 1 line
	Balbir

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#51 lore raw
Quoted 5 lines · On Wed, 25 Mar 2009 08:55:05 +0900 KAMEZAWA Hiroyuki…
Quoted 3 lines · On Tue, 24 Mar 2009 23:04:14 +0530 Balbir Singh <[email protected]>…

I've run lmbench with the soft limit patches and the results show no major overhead, there are some outliers and unexpected results.

Quoted 8 lines
Quoted 4 lines

The outliers are at context-switch 16p/64K, in communicating latencies and some unexpected results where the softlimit changes help improve performance (I consider these to be in the range of noise).

ok, seems no regressions. but what is the softlimit value ? I think there result is of course souftlimit=0 case value...right ?

I'll say no more complains to this hooks even while I don't like them. But res_coutner_charge() looks like decolated chocolate cake as _counter_ ;)

Thanks,
-Kame

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#52 lore raw
Quoted 8 lines · KAMEZAWA Hiroyuki
Quoted 5 lines · On Wed, 25 Mar 2009 08:55:05 +0900 KAMEZAWA Hiroyuki…
Quoted 3 lines · On Tue, 24 Mar 2009 23:04:14 +0530 Balbir Singh <[email protected]>…

I've run lmbench with the soft limit patches and the results show no major overhead, there are some outliers and unexpected results.

Quoted 9 lines
Quoted 8 lines
Quoted 4 lines

The outliers are at context-switch 16p/64K, in communicating latencies and some unexpected results where the softlimit changes help improve performance (I consider these to be in the range of noise).

ok, seems no regressions. but what is the softlimit value ? I think there result is of course souftlimit=0 case value...right ?

Yes, this result is for the soft limit being default value (LONGLONG_MAX) case.

Quoted 3 lines

I'll say no more complains to this hooks even while I don't like them. But res_coutner_charge() looks like decolated chocolate cake as _counter_ ;)

res_counters are split out for modularity reasons, the advantage is that we can optimize/change res_counters without affecting the memcg code. I am glad you can see that there is no overhead as a result of these hooks.

Signature · 7 lines
	Balbir

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
#53 lore raw
Quoted 9 lines · On Wed, 25 Mar 2009 09:32:46 +0530 Balbir Singh <[email protected]>…
Quoted 3 lines

I'll say no more complains to this hooks even while I don't like them. But res_coutner_charge() looks like decolated chocolate cake as _counter_ ;)

res_counters are split out for modularity reasons, the advantage is that we can optimize/change res_counters without affecting the memcg code. I am glad you can see that there is no overhead as a result of these hooks.

I'll use your code in my own set. Anyway, it's merge-window now and we'll have enough time to think of cool stuff.

Thanks,
-Kame

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>