Make res_counter hierarchical

PATCH In review Pavel Emelyanov 8 participants 25 messages Mar 7, 2008 – Apr 3, 2008 2 messages not in the archive

Subsystems (2)

Memory management memcg

lore

Share & tools

lore

Pavel Emelyanov

author

2008-03-07 15:32 UTC #0 lore raw

PATCH 2/2 Make res_counter hierarchical

This allows us two things basically:

1. If the subgroup has the limit higher than its parent has then the one will get more memory than allowed.
2. When we will need to account for a resource in more than one place, we'll be able to use this technics.

Look, consider we have a memory limit and swap limit. The memory limit is the limit for the sum of RSS, page cache and swap usage. To account for this gracefuly, we'll set two counters:

	   res_counter mem_counter;
	   res_counter swap_counter;

attach mm to the swap one

	   mm->mem_cnt = &swap_counter;

and make the swap_counter be mem's child. That's it. If we want hierarchical support, then the tree will look like this:

mem_counter_top
swap_counter_top <- mm_struct living at top
mem_counter_sub
swap_counter_sub <- mm_struct living at sub

Signed-off-by Pavel Emelyanov <[email protected]>

---
 include/linux/res_counter.h |   11 ++++++++++-
 kernel/res_counter.c        |   36 +++++++++++++++++++++++++++++-------
 mm/memcontrol.c             |    9 ++++++---
 3 files changed, 45 insertions(+), 11 deletions(-)

include/linux/res_counter.h +10 -1

index 2c4deb5..a27105e 100644
@@ -41,6 +41,10 @@ struct res_counter {
 	 * the routines below consider this to be IRQ-safe
 	 */
 	spinlock_t lock;
+	/*
+	 * the parent counter. used for hierarchical resource accounting
+	 */
+	struct res_counter *parent;
 };
 
 /**

@@ -80,7 +84,12 @@ enum {
  * helpers for accounting
  */
 
-void res_counter_init(struct res_counter *counter);
+/*
+ * the parent pointer is set only once - during the counter
+ * initialization. caller then must itself provide that this
+ * pointer is valid during the new counter lifetime
+ */
+void res_counter_init(struct res_counter *counter, struct res_counter *parent);
 
 /*
  * charge - try to consume more resource.

kernel/res_counter.c +29 -7

index f1f20c2..046f6f4 100644
@@ -13,10 +13,11 @@
 #include <linux/res_counter.h>
 #include <linux/uaccess.h>
 
-void res_counter_init(struct res_counter *counter)
+void res_counter_init(struct res_counter *counter, struct res_counter *parent)
 {
 	spin_lock_init(&counter->lock);
 	counter->limit = (unsigned long long)LLONG_MAX;
+	counter->parent = parent;
 }
 
 int res_counter_charge_locked(struct res_counter *counter, unsigned long val)

@@ -36,10 +37,26 @@ int res_counter_charge(struct res_counter *counter, unsigned long val)
 {
 	int ret;
 	unsigned long flags;
+	struct res_counter *c, *unroll_c;
+
+	local_irq_save(flags);
+	for (c = counter; c != NULL; c = c->parent) {
+		spin_lock(&c->lock);
+		ret = res_counter_charge_locked(c, val);
+		spin_unlock(&c->lock);
+		if (ret < 0)
+			goto unroll;
+	}
+	local_irq_restore(flags);
+	return 0;
 
-	spin_lock_irqsave(&counter->lock, flags);
-	ret = res_counter_charge_locked(counter, val);
-	spin_unlock_irqrestore(&counter->lock, flags);
+unroll:
+	for (unroll_c = counter; unroll_c != c; unroll_c = unroll_c->parent) {
+		spin_lock(&unroll_c->lock);
+		res_counter_uncharge_locked(unroll_c, val);
+		spin_unlock(&unroll_c->lock);
+	}
+	local_irq_restore(flags);
 	return ret;
 }

@@ -54,10 +71,15 @@ void res_counter_uncharge_locked(struct res_counter *counter, unsigned long val)
 void res_counter_uncharge(struct res_counter *counter, unsigned long val)
 {
 	unsigned long flags;
+	struct res_counter *c;
 
-	spin_lock_irqsave(&counter->lock, flags);
-	res_counter_uncharge_locked(counter, val);
-	spin_unlock_irqrestore(&counter->lock, flags);
+	local_irq_save(flags);
+	for (c = counter; c != NULL; c = c->parent) {
+		spin_lock(&c->lock);
+		res_counter_uncharge_locked(c, val);
+		spin_unlock(&c->lock);
+	}
+	local_irq_restore(flags);
 }

mm/memcontrol.c +6 -3

index e5c741a..61db79c 100644
@@ -976,19 +976,22 @@ static void free_mem_cgroup_per_zone_info(struct mem_cgroup *mem, int node)
 static struct cgroup_subsys_state *
 mem_cgroup_create(struct cgroup_subsys *ss, struct cgroup *cont)
 {
-	struct mem_cgroup *mem;
+	struct mem_cgroup *mem, *parent;
 	int node;
 
 	if (unlikely((cont->parent) == NULL)) {
 		mem = &init_mem_cgroup;
 		init_mm.mem_cgroup = mem;
-	} else
+		parent = NULL;
+	} else {
 		mem = kzalloc(sizeof(struct mem_cgroup), GFP_KERNEL);
+		parent = mem_cgroup_from_cont(cont->parent);
+	}
 
 	if (mem == NULL)
 		return ERR_PTR(-ENOMEM);
 
-	res_counter_init(&mem->res);
+	res_counter_init(&mem->res, parent ? &parent->res : NULL);
 
 	memset(&mem->info, 0, sizeof(mem->info));

Signature · 1 line

1.5.3.4

KAMEZAWA Hiroyuki

2008-03-08 04:45 UTC #1 lore raw

Quoted 19 lines · On Fri, 07 Mar 2008 18:32:20 +0300 Pavel Emelyanov…

This allows us two things basically:
1. If the subgroup has the limit higher than its parent has then the one will get more memory than allowed.
2. When we will need to account for a resource in more than one place, we'll be able to use this technics.
Look, consider we have a memory limit and swap limit. The memory limit is the limit for the sum of RSS, page cache and swap usage. To account for this gracefuly, we'll set two counters:
	   res_counter mem_counter;
	   res_counter swap_counter;
attach mm to the swap one
	   mm->mem_cnt = &swap_counter;

Quoted 8 lines

and make the swap_counter be mem's child. That's it. If we want hierarchical support, then the tree will look like this:
mem_counter_top
swap_counter_top <- mm_struct living at top
mem_counter_sub
swap_counter_sub <- mm_struct living at sub

Hmm? seems strange.

IMO, a parent's usage is just sum of all childs'.
And, historically, memory overcommit is done agaist "memory usage + swap".

How about this ?
<mem_counter_top, swap_counter_top>
<mem_counter_sub, swap_counter_sub>
<mem_counter_sub, swap_counter_sub>
<mem_counter_sub, swap_counter_sub>

mem_counter_top.usage == sum of all mem_coutner_sub.usage swap_counter_sub.usage = sum of all swap_counter_sub.usage

Quoted 14 lines

mm/memcontrol.c +3 -2

@@ -976,19 +976,22 @@ static void free_mem_cgroup_per_zone_info(struct mem_cgroup *mem, int node)
 static struct cgroup_subsys_state *
 mem_cgroup_create(struct cgroup_subsys *ss, struct cgroup *cont)
 {
-	struct mem_cgroup *mem;
+	struct mem_cgroup *mem, *parent;
 	int node;
 
 	if (unlikely((cont->parent) == NULL)) {
 		mem = &init_mem_cgroup;
 		init_mm.mem_cgroup = mem;
-	} else
+		parent = NULL;
+	} else {

mm/memcontrol.c static void free_mem_cgroup_per_zone_info(struct mem_cgroup *mem, int node) · line 995

Quoted 10 lines

mm/memcontrol.c +3 -1

 		mem = kzalloc(sizeof(struct mem_cgroup), GFP_KERNEL);
+		parent = mem_cgroup_from_cont(cont->parent);
+	}
 
 	if (mem == NULL)
 		return ERR_PTR(-ENOMEM);
 
-	res_counter_init(&mem->res);
+	res_counter_init(&mem->res, parent ? &parent->res : NULL);

I have no objection to add some hierarchical support to res_counter.

But we should wait to add it to mem_cgroup because we have to add some amount of codes to handle hierarchy under mem_cgroup in reasonable way. for example)
- hierarchical memory reclaim
- keeping fairness between sub memory controllers. etc...

Thanks,
-Kame

Pavel Emelyanov

author

2008-03-11 08:15 UTC #2 lore raw

Quoted 35 lines · KAMEZAWA Hiroyuki

Quoted 19 lines · On Fri, 07 Mar 2008 18:32:20 +0300 Pavel Emelyanov…
This allows us two things basically:
1. If the subgroup has the limit higher than its parent has then the one will get more memory than allowed.
2. When we will need to account for a resource in more than one place, we'll be able to use this technics.
Look, consider we have a memory limit and swap limit. The memory limit is the limit for the sum of RSS, page cache and swap usage. To account for this gracefuly, we'll set two counters:
	   res_counter mem_counter;
	   res_counter swap_counter;
attach mm to the swap one
	   mm->mem_cnt = &swap_counter;
Quoted 8 lines
and make the swap_counter be mem's child. That's it. If we want hierarchical support, then the tree will look like this:
mem_counter_top
swap_counter_top <- mm_struct living at top
mem_counter_sub
swap_counter_sub <- mm_struct living at sub
Hmm? seems strange.
IMO, a parent's usage is just sum of all childs'.
And, historically, memory overcommit is done agaist "memory usage + swap".

Quoted 8 lines

How about this ?
<mem_counter_top, swap_counter_top>
<mem_counter_sub, swap_counter_sub>
<mem_counter_sub, swap_counter_sub>
<mem_counter_sub, swap_counter_sub>
mem_counter_top.usage == sum of all mem_coutner_sub.usage swap_counter_sub.usage = sum of all swap_counter_sub.usage

I've misprinted in y tree, sorry.
The correct hierarchy as I see it is

<mem_couter_0>
+ -- <swap_counter_0>
+ -- <mem_counter_1>

 |     + -- <swap_counter_1>
 |     + -- <mem_counter_11>
 |     |     + -- <swap_counter_11>
 |     + -- <mem_counter_12>
 |           + -- <swap_counter_12>
 + -- <mem_counter_2>
 |     + -- <swap_counter_2>
 |     + -- <mem_counter_21>
 |     |     + -- <swap_counter_21>
 |     + -- <mem_counter_22>
 |           + -- <swap_counter_22>

+ -- <mem_counter_N>
+ -- <swap_counter_N>
+ -- <mem_counter_N1>
| + -- <swap_counter_N1>
+ -- <mem_counter_N2>
+ -- <swap_counter_N2>

Quoted 38 lines

Quoted 15 lines
mm/memcontrol.c +3 -2
@@ -976,19 +976,22 @@ static void free_mem_cgroup_per_zone_info(struct mem_cgroup *mem, int node)
 static struct cgroup_subsys_state *
 mem_cgroup_create(struct cgroup_subsys *ss, struct cgroup *cont)
 {
-	struct mem_cgroup *mem;
+	struct mem_cgroup *mem, *parent;
 	int node;
 
 	if (unlikely((cont->parent) == NULL)) {
 		mem = &init_mem_cgroup;
 		init_mm.mem_cgroup = mem;
-	} else
+		parent = NULL;
+	} else {
mm/memcontrol.c static void free_mem_cgroup_per_zone_info(struct mem_cgroup *mem, int node) · line 995
Quoted 10 lines
mm/memcontrol.c +3 -1
 		mem = kzalloc(sizeof(struct mem_cgroup), GFP_KERNEL);
+		parent = mem_cgroup_from_cont(cont->parent);
+	}
 
 	if (mem == NULL)
 		return ERR_PTR(-ENOMEM);
 
-	res_counter_init(&mem->res);
+	res_counter_init(&mem->res, parent ? &parent->res : NULL);
 
I have no objection to add some hierarchical support to res_counter.
But we should wait to add it to mem_cgroup because we have to add some amount of codes to handle hierarchy under mem_cgroup in reasonable way. for example)
- hierarchical memory reclaim
- keeping fairness between sub memory controllers. etc...
Thanks,
-Kame

KAMEZAWA Hiroyuki

2008-03-11 08:32 UTC #3 lore raw

Quoted 11 lines · On Tue, 11 Mar 2008 11:15:56 +0300 Pavel Emelyanov…

Quoted 11 lines
Hmm? seems strange.
IMO, a parent's usage is just sum of all childs'.
And, historically, memory overcommit is done agaist "memory usage + swap".
How about this ?
<mem_counter_top, swap_counter_top>
<mem_counter_sub, swap_counter_sub>
<mem_counter_sub, swap_counter_sub>
<mem_counter_sub, swap_counter_sub>

Quoted 6 lines

Quoted 2 lines
mem_counter_top.usage == sum of all mem_coutner_sub.usage swap_counter_sub.usage = sum of all swap_counter_sub.usage
I've misprinted in y tree, sorry.
The correct hierarchy as I see it is

thank you.

Quoted 14 lines

<mem_couter_0>
+ -- <swap_counter_0>
+ -- <mem_counter_1>

 |     + -- <swap_counter_1>
 |     + -- <mem_counter_11>
 |     |     + -- <swap_counter_11>
 |     + -- <mem_counter_12>
 |           + -- <swap_counter_12>
 + -- <mem_counter_2>
 |     + -- <swap_counter_2>
 |     + -- <mem_counter_21>
 |     |     + -- <swap_counter_21>
 |     + -- <mem_counter_22>
 |           + -- <swap_counter_22>

Quoted 7 lines

+ -- <mem_counter_N>
+ -- <swap_counter_N>
+ -- <mem_counter_N1>
| + -- <swap_counter_N1>
+ -- <mem_counter_N2>
+ -- <swap_counter_N2>

please let me confirm.

- swap_counter_X.limit can be defined independent from mem_counter_X.limit ?
- swap_conter_N1's limit and swap_counter_N's have some relationship ?

Thanks,
-kame

Pavel Emelyanov

author

2008-03-11 08:38 UTC #4 lore raw

Quoted 41 lines · KAMEZAWA Hiroyuki

Quoted 11 lines · On Tue, 11 Mar 2008 11:15:56 +0300 Pavel Emelyanov…
Quoted 11 lines
Hmm? seems strange.
IMO, a parent's usage is just sum of all childs'.
And, historically, memory overcommit is done agaist "memory usage + swap".
How about this ?
<mem_counter_top, swap_counter_top>
<mem_counter_sub, swap_counter_sub>
<mem_counter_sub, swap_counter_sub>
<mem_counter_sub, swap_counter_sub>
Quoted 5 lines
Quoted 2 lines
mem_counter_top.usage == sum of all mem_coutner_sub.usage swap_counter_sub.usage = sum of all swap_counter_sub.usage
I've misprinted in y tree, sorry.
The correct hierarchy as I see it is
thank you.
Quoted 14 lines
<mem_couter_0>
+ -- <swap_counter_0>
+ -- <mem_counter_1>
 |     + -- <swap_counter_1>
 |     + -- <mem_counter_11>
 |     |     + -- <swap_counter_11>
 |     + -- <mem_counter_12>
 |           + -- <swap_counter_12>
 + -- <mem_counter_2>
 |     + -- <swap_counter_2>
 |     + -- <mem_counter_21>
 |     |     + -- <swap_counter_21>
 |     + -- <mem_counter_22>
 |           + -- <swap_counter_22>
Quoted 7 lines
+ -- <mem_counter_N>
+ -- <swap_counter_N>
+ -- <mem_counter_N1>
| + -- <swap_counter_N1>
+ -- <mem_counter_N2>
+ -- <swap_counter_N2>

Quoted 4 lines

please let me confirm.
- swap_counter_X.limit can be defined independent from mem_counter_X.limit ?
- swap_conter_N1's limit and swap_counter_N's have some relationship ?

No. The mem_counter_N_limit is the limit for all the memory, that the Nth group consumes. This includes the RSS, page cache and swap for this group and all the child groups. Since RSS and page cache are accounted together, this limit tracks the sum of (memory + swap) values over the subtree started at the given group.

Quoted 4 lines

Thanks,
-kame

Daisuke Nishimura

2008-03-11 09:03 UTC #5 lore raw

Hi.

Quoted 6 lines

No. The mem_counter_N_limit is the limit for all the memory, that the Nth group consumes. This includes the RSS, page cache and swap for this group and all the child groups. Since RSS and page cache are accounted together, this limit tracks the sum of (memory + swap) values over the subtree started at the given group.

It seems a bit confusing for me, because current memcg manages only RSS and page cache, not swap.

Quoted 11 lines

Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
IMO, a parent's usage is just sum of all childs'.
And, historically, memory overcommit is done agaist "memory usage + swap".
How about this ?
<mem_counter_top, swap_counter_top>
<mem_counter_sub, swap_counter_sub>
<mem_counter_sub, swap_counter_sub>
<mem_counter_sub, swap_counter_sub>
mem_counter_top.usage == sum of all mem_coutner_sub.usage swap_counter_sub.usage = sum of all swap_counter_sub.usage

I prefer Kamezawa-san's idea.

Thanks,
Daisuke Nishimura.

KAMEZAWA Hiroyuki

2008-03-11 09:07 UTC #6 lore raw

In reply to Pavel Emelyanov

Quoted 23 lines · On Tue, 11 Mar 2008 11:38:50 +0300 Pavel Emelyanov…

Quoted 23 lines
Quoted 14 lines
<mem_couter_0>
+ -- <swap_counter_0>
+ -- <mem_counter_1>
 |     + -- <swap_counter_1>
 |     + -- <mem_counter_11>
 |     |     + -- <swap_counter_11>
 |     + -- <mem_counter_12>
 |           + -- <swap_counter_12>
 + -- <mem_counter_2>
 |     + -- <swap_counter_2>
 |     + -- <mem_counter_21>
 |     |     + -- <swap_counter_21>
 |     + -- <mem_counter_22>
 |           + -- <swap_counter_22>
Quoted 7 lines
+ -- <mem_counter_N>
+ -- <swap_counter_N>
+ -- <mem_counter_N1>
| + -- <swap_counter_N1>
+ -- <mem_counter_N2>
+ -- <swap_counter_N2>
please let me confirm.

Quoted 9 lines

Quoted 2 lines
- swap_counter_X.limit can be defined independent from mem_counter_X.limit ?
- swap_conter_N1's limit and swap_counter_N's have some relationship ?
No. The mem_counter_N_limit is the limit for all the memory, that the Nth group consumes. This includes the RSS, page cache and swap for this group and all the child groups. Since RSS and page cache are accounted together, this limit tracks the sum of (memory + swap) values over the subtree started at the given group.

Hmm, how should I set limit to allow "tons of swap but small limit to memory".

Thanks,
-Kame

Paul Menage

2008-03-11 08:57 UTC #7 lore raw

In reply to Pavel Emelyanov

Quoted 15 lines · Pavel Emelyanov

<mem_couter_0>
+ -- <swap_counter_0>
+ -- <mem_counter_1>

  |     + -- <swap_counter_1>
  |     + -- <mem_counter_11>
  |     |     + -- <swap_counter_11>
  |     + -- <mem_counter_12>
  |           + -- <swap_counter_12>
  + -- <mem_counter_2>
  |     + -- <swap_counter_2>
  |     + -- <mem_counter_21>
  |     |     + -- <swap_counter_21>
  |     + -- <mem_counter_22>
  |           + -- <swap_counter_22>

Quoted 7 lines

+ -- <mem_counter_N>
+ -- <swap_counter_N>
+ -- <mem_counter_N1>
| + -- <swap_counter_N1>
+ -- <mem_counter_N2>
+ -- <swap_counter_N2>

The idea of hierarchy is good, but I don't think this particular hierarchy works for memory.

Main memory and swap space are very different resources, with very different performance characteristics. Suppose you have a 2G machine, and you want to guarantee each job 1GB of main memory, plus give them the option of 1GB of swap for when they go over the 1G main memory limit. With the hierarchy given above, you've need to give each job a 2GB mem.limit and a 1GB swap.limit, and so there would be no main memory isolation.

My feeling is that people are going to want to limit swap and main memory usage as two independent resource hierarchies more often than they're going to want to limit overall virtual memory. But assuming that there are people who need to do the latter, then you should make it configurable how the hierarchies fit together.

Alternatively, you could make it possible for a res_counter to have multiple parents (each of which constrains the overall usage of it and its siblings), and have three counters for each cgroup:

- vm_counter: overall virtual memory limit for group, parent = parent_mem_cgroup->vm_counter

- mem_counter: main memory limit for group, parents = vm_counter, parent_mem_cgroup->mem_counter

- swap_counter: swap limit for group, parents = vm_counter, parent_mem_cgroup->swap_counter

Paul

KAMEZAWA Hiroyuki

2008-03-11 09:13 UTC #8 lore raw

Quoted 4 lines · On Tue, 11 Mar 2008 01:57:43 -0700 "Paul Menage"…

Alternatively, you could make it possible for a res_counter to have multiple parents (each of which constrains the overall usage of it and its siblings), and have three counters for each cgroup:

Quoted 9 lines

- vm_counter: overall virtual memory limit for group, parent = parent_mem_cgroup->vm_counter
- mem_counter: main memory limit for group, parents = vm_counter, parent_mem_cgroup->mem_counter
- swap_counter: swap limit for group, parents = vm_counter, parent_mem_cgroup->swap_counter

or remove all relationship among counters of *different* type of resources. user-land-daemon will do enough jobs.

Thanks,
-Kame

Paul Menage

2008-03-11 09:11 UTC #9 lore raw

Quoted 3 lines · KAMEZAWA Hiroyuki

or remove all relationship among counters of *different* type of resources. user-land-daemon will do enough jobs.

Yes, that would be my preferred choice, if people agree that hierarchically limiting overall virtual memory isn't useful. (I don't think I have a use for it myself).

Paul

Balbir Singh

2008-03-11 09:16 UTC #10 lore raw

Quoted 10 lines · Paul Menage

Quoted 3 lines · KAMEZAWA Hiroyuki
or remove all relationship among counters of *different* type of resources. user-land-daemon will do enough jobs.
Yes, that would be my preferred choice, if people agree that hierarchically limiting overall virtual memory isn't useful. (I don't think I have a use for it myself).

Virtual limits are very useful. I have a patch ready to send out. They limit the amount of paging a cgroup can do (virtual limit - RSS limit). Some times end users want to set virtual limit == RSS limit, so that the cgroup OOMs on cross the RSS limit.

Signature · 4 lines

	Warm Regards,
	Balbir Singh
	Linux Technology Center
	IBM, ISTL

KAMEZAWA Hiroyuki

2008-03-11 09:39 UTC #11 lore raw

Quoted 12 lines · On Tue, 11 Mar 2008 14:46:58 +0530 Balbir Singh…

Quoted 10 lines · Paul Menage
Quoted 3 lines · KAMEZAWA Hiroyuki
or remove all relationship among counters of *different* type of resources. user-land-daemon will do enough jobs.
Yes, that would be my preferred choice, if people agree that hierarchically limiting overall virtual memory isn't useful. (I don't think I have a use for it myself).

Quoted 5 lines

Virtual limits are very useful. I have a patch ready to send out. They limit the amount of paging a cgroup can do (virtual limit - RSS limit). Some times end users want to set virtual limit == RSS limit, so that the cgroup OOMs on cross the RSS limit.

I have no objection to adding virtual limit itself. (It can be considered as extended ulimit.)

But if you'd like to add relationship between virtual-limit/memory-usage-limit, please take care to make it clear that relationship is reaseonable.

- memory-usage includes page-cache.
- memory-usage doesn't include hugepages.
- How to treat MAP_NORESERVE is depends on over-commit-memory type. how cgroup does ?
- shared memory will be conuted per mmap.

Thanks,
-Kame

Balbir Singh

2008-03-11 09:53 UTC #12 lore raw

Quoted 18 lines · KAMEZAWA Hiroyuki

Quoted 6 lines · On Tue, 11 Mar 2008 14:46:58 +0530 Balbir Singh…
Quoted 5 lines · Paul Menage
Quoted 3 lines · KAMEZAWA Hiroyuki
or remove all relationship among counters of *different* type of resources. user-land-daemon will do enough jobs.
Quoted 9 lines
Quoted 4 lines
Yes, that would be my preferred choice, if people agree that hierarchically limiting overall virtual memory isn't useful. (I don't think I have a use for it myself).
Virtual limits are very useful. I have a patch ready to send out. They limit the amount of paging a cgroup can do (virtual limit - RSS limit). Some times end users want to set virtual limit == RSS limit, so that the cgroup OOMs on cross the RSS limit.

Quoted 6 lines

I have no objection to adding virtual limit itself. (It can be considered as extended ulimit.)
But if you'd like to add relationship between virtual-limit/memory-usage-limit, please take care to make it clear that relationship is reaseonable.

No, I don't want to add a relationship, just plain virtual memory limits and let the system administrators determine what works for them.

Signature · 4 lines

	Warm Regards,
	Balbir Singh
	Linux Technology Center
	IBM, ISTL

KAMEZAWA Hiroyuki

2008-03-11 10:03 UTC #13 lore raw

Quoted 7 lines · On Tue, 11 Mar 2008 15:23:04 +0530 Balbir Singh…

Quoted 3 lines
But if you'd like to add relationship between virtual-limit/memory-usage-limit, please take care to make it clear that relationship is reaseonable.
No, I don't want to add a relationship, just plain virtual memory limits and let the system administrators determine what works for them.

ok :)

Thanks,
-Kame

Paul Menage

2008-03-11 15:56 UTC #14 lore raw

In reply to Balbir Singh

Quoted 8 lines · Balbir Singh

Quoted 6 lines · Paul Menage
Quoted 3 lines · KAMEZAWA Hiroyuki
or remove all relationship among counters of *different* type of resources. user-land-daemon will do enough jobs.

Quoted 7 lines

Quoted 4 lines
Yes, that would be my preferred choice, if people agree that hierarchically limiting overall virtual memory isn't useful. (I don't think I have a use for it myself).
Virtual limits are very useful. I have a patch ready to send out. They limit the amount of paging a cgroup can do (virtual limit - RSS limit).

Ah, from this should I assume that you're talking about virtual address space limits, not virtual memory limits?

My comment above was referring to Pavel's proposal to limit total virtual memory (RAM + swap) for a cgroup, and then limit swap as a subset of that, which basically makes it impossible to limit the RAM usage of cgroups properly if you also want to allow swap usage.

Virtual address space limits are somewhat orthogonal to that.

Paul

Balbir Singh

2008-03-11 15:59 UTC #15 lore raw

Quoted 19 lines · Paul Menage

Quoted 7 lines · Balbir Singh
Quoted 6 lines · Paul Menage
Quoted 3 lines · KAMEZAWA Hiroyuki
or remove all relationship among counters of *different* type of resources. user-land-daemon will do enough jobs.
Quoted 7 lines
Quoted 4 lines
Yes, that would be my preferred choice, if people agree that hierarchically limiting overall virtual memory isn't useful. (I don't think I have a use for it myself).
Virtual limits are very useful. I have a patch ready to send out. They limit the amount of paging a cgroup can do (virtual limit - RSS limit).
Ah, from this should I assume that you're talking about virtual address space limits, not virtual memory limits?

Quoted 7 lines

My comment above was referring to Pavel's proposal to limit total virtual memory (RAM + swap) for a cgroup, and then limit swap as a subset of that, which basically makes it impossible to limit the RAM usage of cgroups properly if you also want to allow swap usage.
Virtual address space limits are somewhat orthogonal to that.

Yes, I was referring to Virtual address limits (along the lines of RLIMIT_AS). I guess it's just confusing terminology. I have patches for Virtual address limits. I should send them out soon.

Signature · 4 lines

	Warm Regards,
	Balbir Singh
	Linux Technology Center
	IBM, ISTL

Balbir Singh

2008-03-08 19:06 UTC #16 lore raw

In reply to Pavel Emelyanov

Quoted 4 lines · Pavel Emelyanov

This allows us two things basically:
1. If the subgroup has the limit higher than its parent has then the one will get more memory than allowed.

But should we allow such configuration? I suspect that we should catch such things at the time of writing the limit.

Quoted 15 lines

2. When we will need to account for a resource in more than one place, we'll be able to use this technics.
Look, consider we have a memory limit and swap limit. The memory limit is the limit for the sum of RSS, page cache and swap usage. To account for this gracefuly, we'll set two counters:
	   res_counter mem_counter;
	   res_counter swap_counter;
attach mm to the swap one
	   mm->mem_cnt = &swap_counter;

Quoted 8 lines

and make the swap_counter be mem's child. That's it. If we want hierarchical support, then the tree will look like this:
mem_counter_top
swap_counter_top <- mm_struct living at top
mem_counter_sub
swap_counter_sub <- mm_struct living at sub

Hmm... not sure about this one. What I want to see is a resource counter hierarchy to mimic the container hierarchy. Then ensure that all limits are set sanely. I am planning to implement shares support on to of resource counters.

Quoted 57 lines

Signed-off-by Pavel Emelyanov <[email protected]>

---
 include/linux/res_counter.h |   11 ++++++++++-
 kernel/res_counter.c        |   36 +++++++++++++++++++++++++++++-------
 mm/memcontrol.c             |    9 ++++++---
 3 files changed, 45 insertions(+), 11 deletions(-)

include/linux/res_counter.h +10 -1

index 2c4deb5..a27105e 100644
@@ -41,6 +41,10 @@ struct res_counter {
 	 * the routines below consider this to be IRQ-safe
 	 */
 	spinlock_t lock;
+	/*
+	 * the parent counter. used for hierarchical resource accounting
+	 */
+	struct res_counter *parent;
 };
 
 /**

@@ -80,7 +84,12 @@ enum {
  * helpers for accounting
  */
 
-void res_counter_init(struct res_counter *counter);
+/*
+ * the parent pointer is set only once - during the counter
+ * initialization. caller then must itself provide that this
+ * pointer is valid during the new counter lifetime
+ */
+void res_counter_init(struct res_counter *counter, struct res_counter *parent);
 
 /*
  * charge - try to consume more resource.

kernel/res_counter.c +2 -1

index f1f20c2..046f6f4 100644
@@ -13,10 +13,11 @@
 #include <linux/res_counter.h>
 #include <linux/uaccess.h>
 
-void res_counter_init(struct res_counter *counter)
+void res_counter_init(struct res_counter *counter, struct res_counter *parent)
 {
 	spin_lock_init(&counter->lock);
 	counter->limit = (unsigned long long)LLONG_MAX;
+	counter->parent = parent;
 }
 
 int res_counter_charge_locked(struct res_counter *counter, unsigned long val)

@@ -36,10 +37,26 @@ int res_counter_charge(struct res_counter *counter, unsigned long val)
 {
 	int ret;

kernel/res_counter.c int res_counter_charge(struct res_counter *counter, unsigned long val) · line 48

Quoted 10 lines

kernel/res_counter.c +9 -0

 	unsigned long flags;
+	struct res_counter *c, *unroll_c;
+
+	local_irq_save(flags);
+	for (c = counter; c != NULL; c = c->parent) {
+		spin_lock(&c->lock);
+		ret = res_counter_charge_locked(c, val);
+		spin_unlock(&c->lock);
+		if (ret < 0)
+			goto unroll;

We'd like to know which resource counter failed to allow charging, so that we can reclaim from that mem_res_cgroup.

Quoted 66 lines

kernel/res_counter.c +18 -6

+	}
+	local_irq_restore(flags);
+	return 0;
 
-	spin_lock_irqsave(&counter->lock, flags);
-	ret = res_counter_charge_locked(counter, val);
-	spin_unlock_irqrestore(&counter->lock, flags);
+unroll:
+	for (unroll_c = counter; unroll_c != c; unroll_c = unroll_c->parent) {
+		spin_lock(&unroll_c->lock);
+		res_counter_uncharge_locked(unroll_c, val);
+		spin_unlock(&unroll_c->lock);
+	}
+	local_irq_restore(flags);
 	return ret;
 }

@@ -54,10 +71,15 @@ void res_counter_uncharge_locked(struct res_counter *counter, unsigned long val)
 void res_counter_uncharge(struct res_counter *counter, unsigned long val)
 {
 	unsigned long flags;
+	struct res_counter *c;
 
-	spin_lock_irqsave(&counter->lock, flags);
-	res_counter_uncharge_locked(counter, val);
-	spin_unlock_irqrestore(&counter->lock, flags);
+	local_irq_save(flags);
+	for (c = counter; c != NULL; c = c->parent) {
+		spin_lock(&c->lock);
+		res_counter_uncharge_locked(c, val);
+		spin_unlock(&c->lock);
+	}
+	local_irq_restore(flags);
 }

mm/memcontrol.c +6 -3

index e5c741a..61db79c 100644
@@ -976,19 +976,22 @@ static void free_mem_cgroup_per_zone_info(struct mem_cgroup *mem, int node)
 static struct cgroup_subsys_state *
 mem_cgroup_create(struct cgroup_subsys *ss, struct cgroup *cont)
 {
-	struct mem_cgroup *mem;
+	struct mem_cgroup *mem, *parent;
 	int node;
 
 	if (unlikely((cont->parent) == NULL)) {
 		mem = &init_mem_cgroup;
 		init_mm.mem_cgroup = mem;
-	} else
+		parent = NULL;
+	} else {
 		mem = kzalloc(sizeof(struct mem_cgroup), GFP_KERNEL);
+		parent = mem_cgroup_from_cont(cont->parent);
+	}
 
 	if (mem == NULL)
 		return ERR_PTR(-ENOMEM);
 
-	res_counter_init(&mem->res);
+	res_counter_init(&mem->res, parent ? &parent->res : NULL);
 
 	memset(&mem->info, 0, sizeof(mem->info));

Signature · 4 lines

	Warm Regards,
	Balbir Singh
	Linux Technology Center
	IBM, ISTL

Pavel Emelyanov

author

2008-03-11 08:17 UTC #17 lore raw

Quoted 8 lines · Balbir Singh

Quoted 4 lines · Pavel Emelyanov
This allows us two things basically:
1. If the subgroup has the limit higher than its parent has then the one will get more memory than allowed.
But should we allow such configuration? I suspect that we should catch such things at the time of writing the limit.

We cannot catch this at the limit-set-time. See, if you have a cgroup A with a 1GB limit and the usage is 999Mb, then creating a subgroup B with even 500MB limit will cause the A group consume 1.5GB of memory effectively.

Quoted 168 lines

Quoted 15 lines

2. When we will need to account for a resource in more than one place, we'll be able to use this technics.
Look, consider we have a memory limit and swap limit. The memory limit is the limit for the sum of RSS, page cache and swap usage. To account for this gracefuly, we'll set two counters:
	   res_counter mem_counter;
	   res_counter swap_counter;
attach mm to the swap one
	   mm->mem_cnt = &swap_counter;

Quoted 8 lines

and make the swap_counter be mem's child. That's it. If we want hierarchical support, then the tree will look like this:
mem_counter_top
swap_counter_top <- mm_struct living at top
mem_counter_sub
swap_counter_sub <- mm_struct living at sub

Quoted 57 lines

Signed-off-by Pavel Emelyanov <[email protected]>

---
 include/linux/res_counter.h |   11 ++++++++++-
 kernel/res_counter.c        |   36 +++++++++++++++++++++++++++++-------
 mm/memcontrol.c             |    9 ++++++---
 3 files changed, 45 insertions(+), 11 deletions(-)

include/linux/res_counter.h +10 -1

index 2c4deb5..a27105e 100644
@@ -41,6 +41,10 @@ struct res_counter {
 	 * the routines below consider this to be IRQ-safe
 	 */
 	spinlock_t lock;
+	/*
+	 * the parent counter. used for hierarchical resource accounting
+	 */
+	struct res_counter *parent;
 };
 
 /**

@@ -80,7 +84,12 @@ enum {
  * helpers for accounting
  */
 
-void res_counter_init(struct res_counter *counter);
+/*
+ * the parent pointer is set only once - during the counter
+ * initialization. caller then must itself provide that this
+ * pointer is valid during the new counter lifetime
+ */
+void res_counter_init(struct res_counter *counter, struct res_counter *parent);
 
 /*
  * charge - try to consume more resource.

kernel/res_counter.c +2 -1

index f1f20c2..046f6f4 100644
@@ -13,10 +13,11 @@
 #include <linux/res_counter.h>
 #include <linux/uaccess.h>
 
-void res_counter_init(struct res_counter *counter)
+void res_counter_init(struct res_counter *counter, struct res_counter *parent)
 {
 	spin_lock_init(&counter->lock);
 	counter->limit = (unsigned long long)LLONG_MAX;
+	counter->parent = parent;
 }
 
 int res_counter_charge_locked(struct res_counter *counter, unsigned long val)

@@ -36,10 +37,26 @@ int res_counter_charge(struct res_counter *counter, unsigned long val)
 {
 	int ret;

kernel/res_counter.c int res_counter_charge(struct res_counter *counter, unsigned long val) · line 48

Quoted 10 lines

kernel/res_counter.c +9 -0

 	unsigned long flags;
+	struct res_counter *c, *unroll_c;
+
+	local_irq_save(flags);
+	for (c = counter; c != NULL; c = c->parent) {
+		spin_lock(&c->lock);
+		ret = res_counter_charge_locked(c, val);
+		spin_unlock(&c->lock);
+		if (ret < 0)
+			goto unroll;

We'd like to know which resource counter failed to allow charging, so that we can reclaim from that mem_res_cgroup.

Quoted 66 lines

kernel/res_counter.c +18 -6

+	}
+	local_irq_restore(flags);
+	return 0;
 
-	spin_lock_irqsave(&counter->lock, flags);
-	ret = res_counter_charge_locked(counter, val);
-	spin_unlock_irqrestore(&counter->lock, flags);
+unroll:
+	for (unroll_c = counter; unroll_c != c; unroll_c = unroll_c->parent) {
+		spin_lock(&unroll_c->lock);
+		res_counter_uncharge_locked(unroll_c, val);
+		spin_unlock(&unroll_c->lock);
+	}
+	local_irq_restore(flags);
 	return ret;
 }

@@ -54,10 +71,15 @@ void res_counter_uncharge_locked(struct res_counter *counter, unsigned long val)
 void res_counter_uncharge(struct res_counter *counter, unsigned long val)
 {
 	unsigned long flags;
+	struct res_counter *c;
 
-	spin_lock_irqsave(&counter->lock, flags);
-	res_counter_uncharge_locked(counter, val);
-	spin_unlock_irqrestore(&counter->lock, flags);
+	local_irq_save(flags);
+	for (c = counter; c != NULL; c = c->parent) {
+		spin_lock(&c->lock);
+		res_counter_uncharge_locked(c, val);
+		spin_unlock(&c->lock);
+	}
+	local_irq_restore(flags);
 }

mm/memcontrol.c +6 -3

index e5c741a..61db79c 100644
@@ -976,19 +976,22 @@ static void free_mem_cgroup_per_zone_info(struct mem_cgroup *mem, int node)
 static struct cgroup_subsys_state *
 mem_cgroup_create(struct cgroup_subsys *ss, struct cgroup *cont)
 {
-	struct mem_cgroup *mem;
+	struct mem_cgroup *mem, *parent;
 	int node;
 
 	if (unlikely((cont->parent) == NULL)) {
 		mem = &init_mem_cgroup;
 		init_mm.mem_cgroup = mem;
-	} else
+		parent = NULL;
+	} else {
 		mem = kzalloc(sizeof(struct mem_cgroup), GFP_KERNEL);
+		parent = mem_cgroup_from_cont(cont->parent);
+	}
 
 	if (mem == NULL)
 		return ERR_PTR(-ENOMEM);
 
-	res_counter_init(&mem->res);
+	res_counter_init(&mem->res, parent ? &parent->res : NULL);
 
 	memset(&mem->info, 0, sizeof(mem->info));

Balbir Singh

2008-03-11 08:24 UTC #18 lore raw

Quoted 6 lines · Pavel Emelyanov

Quoted 5 lines · Balbir Singh
Quoted 4 lines · Pavel Emelyanov
This allows us two things basically:
1. If the subgroup has the limit higher than its parent has then the one will get more memory than allowed.

Quoted 8 lines

Quoted 2 lines
But should we allow such configuration? I suspect that we should catch such things at the time of writing the limit.
We cannot catch this at the limit-set-time. See, if you have a cgroup A with a 1GB limit and the usage is 999Mb, then creating a subgroup B with even 500MB limit will cause the A group consume 1.5GB of memory effectively.

No... If you propagate the charge of the child up to the parent, then it won't. If each page charged to a child is also charged to the parent, this cannot happen. The code you have below does that right?

Quoted 90 lines

Quoted 90 lines
Quoted 15 lines
2. When we will need to account for a resource in more than one place, we'll be able to use this technics.
Look, consider we have a memory limit and swap limit. The memory limit is the limit for the sum of RSS, page cache and swap usage. To account for this gracefuly, we'll set two counters:
	   res_counter mem_counter;
	   res_counter swap_counter;
attach mm to the swap one
	   mm->mem_cnt = &swap_counter;
Quoted 8 lines
and make the swap_counter be mem's child. That's it. If we want hierarchical support, then the tree will look like this:
mem_counter_top
swap_counter_top <- mm_struct living at top
mem_counter_sub
swap_counter_sub <- mm_struct living at sub
Hmm... not sure about this one. What I want to see is a resource counter hierarchy to mimic the container hierarchy. Then ensure that all limits are set sanely. I am planning to implement shares support on to of resource counters.
Quoted 62 lines
Signed-off-by Pavel Emelyanov <[email protected]>
---
 include/linux/res_counter.h |   11 ++++++++++-
 kernel/res_counter.c        |   36 +++++++++++++++++++++++++++++-------
 mm/memcontrol.c             |    9 ++++++---
 3 files changed, 45 insertions(+), 11 deletions(-)
include/linux/res_counter.h +10 -1
index 2c4deb5..a27105e 100644
@@ -41,6 +41,10 @@ struct res_counter {
 	 * the routines below consider this to be IRQ-safe
 	 */
 	spinlock_t lock;
+	/*
+	 * the parent counter. used for hierarchical resource accounting
+	 */
+	struct res_counter *parent;
 };
 
 /**
@@ -80,7 +84,12 @@ enum {
  * helpers for accounting
  */
 
-void res_counter_init(struct res_counter *counter);
+/*
+ * the parent pointer is set only once - during the counter
+ * initialization. caller then must itself provide that this
+ * pointer is valid during the new counter lifetime
+ */
+void res_counter_init(struct res_counter *counter, struct res_counter *parent);
 
 /*
  * charge - try to consume more resource.
kernel/res_counter.c +6 -1
index f1f20c2..046f6f4 100644
@@ -13,10 +13,11 @@
 #include <linux/res_counter.h>
 #include <linux/uaccess.h>
 
-void res_counter_init(struct res_counter *counter)
+void res_counter_init(struct res_counter *counter, struct res_counter *parent)
 {
 	spin_lock_init(&counter->lock);
 	counter->limit = (unsigned long long)LLONG_MAX;
+	counter->parent = parent;
 }
 
 int res_counter_charge_locked(struct res_counter *counter, unsigned long val)
@@ -36,10 +37,26 @@ int res_counter_charge(struct res_counter *counter, unsigned long val)
 {
 	int ret;
 	unsigned long flags;
+	struct res_counter *c, *unroll_c;
+
+	local_irq_save(flags);
+	for (c = counter; c != NULL; c = c->parent) {

Quoted 8 lines

Quoted 8 lines
kernel/res_counter.c int res_counter_charge(struct res_counter *counter, unsigned long val) · line 48
Quoted 5 lines
kernel/res_counter.c +5 -0
+		spin_lock(&c->lock);
+		ret = res_counter_charge_locked(c, val);
+		spin_unlock(&c->lock);
+		if (ret < 0)
+			goto unroll;
We'd like to know which resource counter failed to allow charging, so that we can reclaim from that mem_res_cgroup.

This is also important, so that we can reclaim from the nodes that go over their limit.

Quoted 73 lines

Quoted 58 lines

kernel/res_counter.c +18 -6

+	}
+	local_irq_restore(flags);
+	return 0;
 
-	spin_lock_irqsave(&counter->lock, flags);
-	ret = res_counter_charge_locked(counter, val);
-	spin_unlock_irqrestore(&counter->lock, flags);
+unroll:
+	for (unroll_c = counter; unroll_c != c; unroll_c = unroll_c->parent) {
+		spin_lock(&unroll_c->lock);
+		res_counter_uncharge_locked(unroll_c, val);
+		spin_unlock(&unroll_c->lock);
+	}
+	local_irq_restore(flags);
 	return ret;
 }

@@ -54,10 +71,15 @@ void res_counter_uncharge_locked(struct res_counter *counter, unsigned long val)
 void res_counter_uncharge(struct res_counter *counter, unsigned long val)
 {
 	unsigned long flags;
+	struct res_counter *c;
 
-	spin_lock_irqsave(&counter->lock, flags);
-	res_counter_uncharge_locked(counter, val);
-	spin_unlock_irqrestore(&counter->lock, flags);
+	local_irq_save(flags);
+	for (c = counter; c != NULL; c = c->parent) {
+		spin_lock(&c->lock);
+		res_counter_uncharge_locked(c, val);
+		spin_unlock(&c->lock);
+	}
+	local_irq_restore(flags);
 }

mm/memcontrol.c +5 -2

index e5c741a..61db79c 100644
@@ -976,19 +976,22 @@ static void free_mem_cgroup_per_zone_info(struct mem_cgroup *mem, int node)
 static struct cgroup_subsys_state *
 mem_cgroup_create(struct cgroup_subsys *ss, struct cgroup *cont)
 {
-	struct mem_cgroup *mem;
+	struct mem_cgroup *mem, *parent;
 	int node;
 
 	if (unlikely((cont->parent) == NULL)) {
 		mem = &init_mem_cgroup;
 		init_mm.mem_cgroup = mem;
-	} else
+		parent = NULL;
+	} else {
 		mem = kzalloc(sizeof(struct mem_cgroup), GFP_KERNEL);
+		parent = mem_cgroup_from_cont(cont->parent);
+	}

Quoted 9 lines

mm/memcontrol.c static void free_mem_cgroup_per_zone_info(struct mem_cgroup *mem, int node) · line 997
Quoted 8 lines
mm/memcontrol.c +1 -1
 	if (mem == NULL)
 		return ERR_PTR(-ENOMEM);
 
-	res_counter_init(&mem->res);
+	res_counter_init(&mem->res, parent ? &parent->res : NULL);
 
 	memset(&mem->info, 0, sizeof(mem->info));
 

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected] For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Signature · 4 lines

	Warm Regards,
	Balbir Singh
	Linux Technology Center
	IBM, ISTL

Pavel Emelyanov

author

2008-03-11 08:40 UTC #19 lore raw

Quoted 15 lines · Balbir Singh

Quoted 4 lines · Pavel Emelyanov
Quoted 3 lines · Balbir Singh
Quoted 2 lines · Pavel Emelyanov
This allows us two things basically:
Quoted 9 lines
Quoted 4 lines
Quoted 2 lines
1. If the subgroup has the limit higher than its parent has then the one will get more memory than allowed.
But should we allow such configuration? I suspect that we should catch such things at the time of writing the limit.
We cannot catch this at the limit-set-time. See, if you have a cgroup A with a 1GB limit and the usage is 999Mb, then creating a subgroup B with even 500MB limit will cause the A group consume 1.5GB of memory effectively.

Quoted 3 lines

No... If you propagate the charge of the child up to the parent, then it won't. If each page charged to a child is also charged to the parent, this cannot happen. The code you have below does that right?

Yup! What you described is available with this patch only.

Quoted 99 lines

Quoted 90 lines
Quoted 90 lines
Quoted 15 lines
2. When we will need to account for a resource in more than one place, we'll be able to use this technics.
Look, consider we have a memory limit and swap limit. The memory limit is the limit for the sum of RSS, page cache and swap usage. To account for this gracefuly, we'll set two counters:
	   res_counter mem_counter;
	   res_counter swap_counter;
attach mm to the swap one
	   mm->mem_cnt = &swap_counter;
Quoted 8 lines
and make the swap_counter be mem's child. That's it. If we want hierarchical support, then the tree will look like this:
mem_counter_top
swap_counter_top <- mm_struct living at top
mem_counter_sub
swap_counter_sub <- mm_struct living at sub
Hmm... not sure about this one. What I want to see is a resource counter hierarchy to mimic the container hierarchy. Then ensure that all limits are set sanely. I am planning to implement shares support on to of resource counters.
Quoted 62 lines
Signed-off-by Pavel Emelyanov <[email protected]>
---
 include/linux/res_counter.h |   11 ++++++++++-
 kernel/res_counter.c        |   36 +++++++++++++++++++++++++++++-------
 mm/memcontrol.c             |    9 ++++++---
 3 files changed, 45 insertions(+), 11 deletions(-)
include/linux/res_counter.h +10 -1
index 2c4deb5..a27105e 100644
@@ -41,6 +41,10 @@ struct res_counter {
 	 * the routines below consider this to be IRQ-safe
 	 */
 	spinlock_t lock;
+	/*
+	 * the parent counter. used for hierarchical resource accounting
+	 */
+	struct res_counter *parent;
 };
 
 /**
@@ -80,7 +84,12 @@ enum {
  * helpers for accounting
  */
 
-void res_counter_init(struct res_counter *counter);
+/*
+ * the parent pointer is set only once - during the counter
+ * initialization. caller then must itself provide that this
+ * pointer is valid during the new counter lifetime
+ */
+void res_counter_init(struct res_counter *counter, struct res_counter *parent);
 
 /*
  * charge - try to consume more resource.
kernel/res_counter.c +6 -1
index f1f20c2..046f6f4 100644
@@ -13,10 +13,11 @@
 #include <linux/res_counter.h>
 #include <linux/uaccess.h>
 
-void res_counter_init(struct res_counter *counter)
+void res_counter_init(struct res_counter *counter, struct res_counter *parent)
 {
 	spin_lock_init(&counter->lock);
 	counter->limit = (unsigned long long)LLONG_MAX;
+	counter->parent = parent;
 }
 
 int res_counter_charge_locked(struct res_counter *counter, unsigned long val)
@@ -36,10 +37,26 @@ int res_counter_charge(struct res_counter *counter, unsigned long val)
 {
 	int ret;
 	unsigned long flags;
+	struct res_counter *c, *unroll_c;
+
+	local_irq_save(flags);
+	for (c = counter; c != NULL; c = c->parent) {
Quoted 8 lines
Quoted 8 lines
kernel/res_counter.c int res_counter_charge(struct res_counter *counter, unsigned long val) · line 48
Quoted 5 lines
kernel/res_counter.c +5 -0
+		spin_lock(&c->lock);
+		ret = res_counter_charge_locked(c, val);
+		spin_unlock(&c->lock);
+		if (ret < 0)
+			goto unroll;
We'd like to know which resource counter failed to allow charging, so that we can reclaim from that mem_res_cgroup.

Quoted 2 lines

This is also important, so that we can reclaim from the nodes that go over their limit.

Agree. I'll think over how to provide this facility.

Quoted 73 lines

Quoted 71 lines

Quoted 58 lines

kernel/res_counter.c +18 -6

+	}
+	local_irq_restore(flags);
+	return 0;
 
-	spin_lock_irqsave(&counter->lock, flags);
-	ret = res_counter_charge_locked(counter, val);
-	spin_unlock_irqrestore(&counter->lock, flags);
+unroll:
+	for (unroll_c = counter; unroll_c != c; unroll_c = unroll_c->parent) {
+		spin_lock(&unroll_c->lock);
+		res_counter_uncharge_locked(unroll_c, val);
+		spin_unlock(&unroll_c->lock);
+	}
+	local_irq_restore(flags);
 	return ret;
 }

@@ -54,10 +71,15 @@ void res_counter_uncharge_locked(struct res_counter *counter, unsigned long val)
 void res_counter_uncharge(struct res_counter *counter, unsigned long val)
 {
 	unsigned long flags;
+	struct res_counter *c;
 
-	spin_lock_irqsave(&counter->lock, flags);
-	res_counter_uncharge_locked(counter, val);
-	spin_unlock_irqrestore(&counter->lock, flags);
+	local_irq_save(flags);
+	for (c = counter; c != NULL; c = c->parent) {
+		spin_lock(&c->lock);
+		res_counter_uncharge_locked(c, val);
+		spin_unlock(&c->lock);
+	}
+	local_irq_restore(flags);
 }

mm/memcontrol.c +5 -2

index e5c741a..61db79c 100644
@@ -976,19 +976,22 @@ static void free_mem_cgroup_per_zone_info(struct mem_cgroup *mem, int node)
 static struct cgroup_subsys_state *
 mem_cgroup_create(struct cgroup_subsys *ss, struct cgroup *cont)
 {
-	struct mem_cgroup *mem;
+	struct mem_cgroup *mem, *parent;
 	int node;
 
 	if (unlikely((cont->parent) == NULL)) {
 		mem = &init_mem_cgroup;
 		init_mm.mem_cgroup = mem;
-	} else
+		parent = NULL;
+	} else {
 		mem = kzalloc(sizeof(struct mem_cgroup), GFP_KERNEL);
+		parent = mem_cgroup_from_cont(cont->parent);
+	}

Quoted 8 lines

mm/memcontrol.c static void free_mem_cgroup_per_zone_info(struct mem_cgroup *mem, int node) · line 997
Quoted 8 lines
mm/memcontrol.c +1 -1
 	if (mem == NULL)
 		return ERR_PTR(-ENOMEM);
 
-	res_counter_init(&mem->res);
+	res_counter_init(&mem->res, parent ? &parent->res : NULL);
 
 	memset(&mem->info, 0, sizeof(mem->info));
 

YAMAMOTO Takashi

2008-03-12 23:05 UTC #20 lore raw

In reply to Pavel Emelyanov

Quoted 19 lines

kernel/res_counter.c +12 -2

@@ -36,10 +37,26 @@ int res_counter_charge(struct res_counter *counter, unsigned long val)
 {
 	int ret;
 	unsigned long flags;
+	struct res_counter *c, *unroll_c;
+
+	local_irq_save(flags);
+	for (c = counter; c != NULL; c = c->parent) {
+		spin_lock(&c->lock);
+		ret = res_counter_charge_locked(c, val);
+		spin_unlock(&c->lock);
+		if (ret < 0)
+			goto unroll;
+	}
+	local_irq_restore(flags);
+	return 0;
 
-	spin_lock_irqsave(&counter->lock, flags);
-	ret = res_counter_charge_locked(counter, val);

kernel/res_counter.c int res_counter_charge(struct res_counter *counter, unsigned long val) · line 61

Quoted 10 lines

kernel/res_counter.c +7 -1

-	spin_unlock_irqrestore(&counter->lock, flags);
+unroll:
+	for (unroll_c = counter; unroll_c != c; unroll_c = unroll_c->parent) {
+		spin_lock(&unroll_c->lock);
+		res_counter_uncharge_locked(unroll_c, val);
+		spin_unlock(&unroll_c->lock);
+	}
+	local_irq_restore(flags);
 	return ret;
 }

what prevents the topology (in particular, ->parent pointers) from changing behind us?

YAMAMOTO Takashi

2008-03-12 23:36 UTC #21 lore raw

Quoted 33 lines

Quoted 19 lines

kernel/res_counter.c +12 -2

@@ -36,10 +37,26 @@ int res_counter_charge(struct res_counter *counter, unsigned long val)
 {
 	int ret;
 	unsigned long flags;
+	struct res_counter *c, *unroll_c;
+
+	local_irq_save(flags);
+	for (c = counter; c != NULL; c = c->parent) {
+		spin_lock(&c->lock);
+		ret = res_counter_charge_locked(c, val);
+		spin_unlock(&c->lock);
+		if (ret < 0)
+			goto unroll;
+	}
+	local_irq_restore(flags);
+	return 0;
 
-	spin_lock_irqsave(&counter->lock, flags);
-	ret = res_counter_charge_locked(counter, val);

kernel/res_counter.c int res_counter_charge(struct res_counter *counter, unsigned long val) · line 61

Quoted 10 lines

kernel/res_counter.c +7 -1

-	spin_unlock_irqrestore(&counter->lock, flags);
+unroll:
+	for (unroll_c = counter; unroll_c != c; unroll_c = unroll_c->parent) {
+		spin_lock(&unroll_c->lock);
+		res_counter_uncharge_locked(unroll_c, val);
+		spin_unlock(&unroll_c->lock);
+	}
+	local_irq_restore(flags);
 	return ret;
 }

what prevents the topology (in particular, ->parent pointers) from changing behind us?

Quoted 1 line

YAMAMOTO Takashi

to answer myself: cgroupfs rename doesn't allow topological changes in the first place.

btw, i think you need to do the same for res_counter_limit_check_locked as well. i'm skeptical about doing these complicated stuffs in kernel, esp. in this potentially performance critical code.

YAMAMOTO Takashi

Pavel Emelyanov

author

2008-03-13 08:56 UTC #22 lore raw

In reply to YAMAMOTO Takashi

Quoted 17 lines · YAMAMOTO Takashi

Quoted 17 lines

kernel/res_counter.c +12 -0

@@ -36,10 +37,26 @@ int res_counter_charge(struct res_counter *counter, unsigned long val)
 {
 	int ret;
 	unsigned long flags;
+	struct res_counter *c, *unroll_c;
+
+	local_irq_save(flags);
+	for (c = counter; c != NULL; c = c->parent) {
+		spin_lock(&c->lock);
+		ret = res_counter_charge_locked(c, val);
+		spin_unlock(&c->lock);
+		if (ret < 0)
+			goto unroll;
+	}
+	local_irq_restore(flags);
+	return 0;

Quoted 15 lines

kernel/res_counter.c int res_counter_charge(struct res_counter *counter, unsigned long val) · line 61
Quoted 12 lines
kernel/res_counter.c +7 -3
-	spin_lock_irqsave(&counter->lock, flags);
-	ret = res_counter_charge_locked(counter, val);
-	spin_unlock_irqrestore(&counter->lock, flags);
+unroll:
+	for (unroll_c = counter; unroll_c != c; unroll_c = unroll_c->parent) {
+		spin_lock(&unroll_c->lock);
+		res_counter_uncharge_locked(unroll_c, val);
+		spin_unlock(&unroll_c->lock);
+	}
+	local_irq_restore(flags);
 	return ret;
 }
what prevents the topology (in particular, ->parent pointers) from changing behind us?

The res_counter client must provide this. Currently cgroup subsystem does this.

Quoted 2 lines

YAMAMOTO Takashi

A message that is not in the archive (sent off-list or not yet received): [email protected]

Balbir Singh

2008-04-02 15:26 UTC #24 lore raw

Quoted 2 lines · Pavel Emelyanov

This allows us two things basically:

Pavel,

Do you have any further updates on this. I think we need a way of being able to implement reclaim per hierarchy as mentioned earlier. Do you want me to take a look at it?

Signature · 4 lines

	Warm Regards,
	Balbir Singh
	Linux Technology Center
	IBM, ISTL

A message that is not in the archive (sent off-list or not yet received): 47F3A5BF.1080301-23VcF4HTsmIX0ybBhKVfKdBPR1lH4CV8@public.gmane.org

Pavel Emelyanov

2008-04-03 12:26 UTC #26 lore raw

Quoted 7 lines · Balbir Singh

Quoted 2 lines · Pavel Emelyanov
This allows us two things basically:
Pavel,
Do you have any further updates on this. I think we need a way of being able to

No. Unfortunately I stopped following the discussion at some point and decided that nobody liked this patch that much.

Quoted 2 lines

implement reclaim per hierarchy as mentioned earlier. Do you want me to take a look at it?

Yes, sure. I'm now busy (among other stuff) with kmemsize controller, hope I can finish its polishing and testing till summer :(