mm: memory.low heirarchical behavior

This patch aims to address an issue in current memory.low semantics, which makes it hard to use it in a hierarchy, where some leaf memory cgroups are more valuable than others.

For example, there are memcgs A, A/B, A/C, A/D and A/E:

A A/memory.low = 2G, A/memory.current = 6G
//\\

BC  DE   B/memory.low = 3G  B/memory.usage = 2G
         C/memory.low = 1G  C/memory.usage = 2G
         D/memory.low = 0   D/memory.usage = 2G

E/memory.low = 10G E/memory.usage = 0

If we apply memory pressure, B, C and D are reclaimed at the same pace while A's usage exceeds 2G.
This is obviously wrong, as B's usage is fully below B's memory.low, and C has 1G of protection as well.
Also, A is pushed to the size, which is less than A's 2G memory.low, which is also wrong.

A simple bash script (provided below) can be used to reproduce the problem. Current results are:

  A:    1430097920
  A/B:  711929856
  A/C:  717426688
  A/D:  741376
  A/E:  0

To address the issue a concept of effective memory.low is introduced. Effective memory.low is always equal or less than original memory.low. In a case, when there is no memory.low overcommittment (and also for top-level cgroups), these two values are equal.
Otherwise it's a part of parent's effective memory.low, calculated as a cgroup's memory.low usage divided by sum of sibling's memory.low usages (under memory.low usage I mean the size of actually protected
memory: memory.current if memory.current < memory.low, 0 otherwise).

It's necessary to track the actual usage, because otherwise an empty cgroup with memory.low set (A/E in my example) will affect actual memory distribution, which makes no sense.

Effective memory.low is always capped by memory.low, set by user. That means it's not possible to become a larger guarantee than memory.low set by a user, even if corresponding part of parent's guarantee is larger. This matches existing semantics.

Calculating effective memory.low can be done in the reclaim path, as we conveniently traversing the cgroup tree from top to bottom and check memory.low on each level. So, it's a perfect place to calculate effective memory low and save it to use it for children cgroups.

This also eliminates a need to traverse the cgroup tree from bottom to top each time to check if parent's guarantee is not exceeded.

Setting/resetting effective memory.low is intentionally racy, but it's fine and shouldn't lead to any significant differences in actual memory distribution.

With this patch applied results are matching the expectations:

  A:    2146160640
  A/B:  1427795968
  A/C:  717705216
  A/D:  659456
  A/E:  0

mm/memcontrol.c +19 -0

@@ -1672,6 +1672,36 @@ void unlock_page_memcg(struct page *page)
 }
 EXPORT_SYMBOL(unlock_page_memcg);
 
+static void memcg_update_low(struct mem_cgroup *memcg)
+{
+	unsigned long usage, low_usage, prev_low_usage;
+	struct mem_cgroup *parent;
+	long delta;
+
+	do {
+		parent = parent_mem_cgroup(memcg);
+		if (!parent || mem_cgroup_is_root(parent))
+			break;
+
+		if (!memcg->low && !atomic_long_read(&memcg->low_usage))
+			break;
+
+		usage = page_counter_read(&memcg->memory);
+		if (usage < memcg->low)
+			low_usage = usage;
+		else
+			low_usage = 0;

Hi Roman,

Quoted 20 lines · Roman Gushchin

This patch aims to address an issue in current memory.low semantics, which makes it hard to use it in a hierarchy, where some leaf memory cgroups are more valuable than others.

For example, there are memcgs A, A/B, A/C, A/D and A/E:

A A/memory.low = 2G, A/memory.current = 6G
//\\

BC  DE   B/memory.low = 3G  B/memory.usage = 2G
         C/memory.low = 1G  C/memory.usage = 2G
         D/memory.low = 0   D/memory.usage = 2G

E/memory.low = 10G E/memory.usage = 0

If we apply memory pressure, B, C and D are reclaimed at the same pace while A's usage exceeds 2G.
This is obviously wrong, as B's usage is fully below B's memory.low, and C has 1G of protection as well.
Also, A is pushed to the size, which is less than A's 2G memory.low, which is also wrong.

Quoted 7 lines

A simple bash script (provided below) can be used to reproduce the problem. Current results are:

  A:    1430097920
  A/B:  711929856
  A/C:  717426688
  A/D:  741376
  A/E:  0

Yes, this is a problem. And the behavior with your patch looks much preferable over the status quo.

Quoted 8 lines

To address the issue a concept of effective memory.low is introduced. Effective memory.low is always equal or less than original memory.low. In a case, when there is no memory.low overcommittment (and also for top-level cgroups), these two values are equal.
Otherwise it's a part of parent's effective memory.low, calculated as a cgroup's memory.low usage divided by sum of sibling's memory.low usages (under memory.low usage I mean the size of actually protected
memory: memory.current if memory.current < memory.low, 0 otherwise).

This hurts my brain.

Why is memory.current == memory.low (which should fully protect memory.current) a low usage of 0?

Why is memory.current > memory.low not a low usage of memory.low?

I.e. shouldn't this be low_usage = min(memory.current, memory.low)?

Quoted 4 lines

It's necessary to track the actual usage, because otherwise an empty cgroup with memory.low set (A/E in my example) will affect actual memory distribution, which makes no sense.

Yep, that makes sense.

Quoted 4 lines

Effective memory.low is always capped by memory.low, set by user. That means it's not possible to become a larger guarantee than memory.low set by a user, even if corresponding part of parent's guarantee is larger. This matches existing semantics.

That's a complicated sentence for an intuitive concept: yes, we wouldn't expect a group's protected usage to exceed its own memory.low setting just because the parent's is higher. I'd drop this part.

Quoted 12 lines

Calculating effective memory.low can be done in the reclaim path, as we conveniently traversing the cgroup tree from top to bottom and check memory.low on each level. So, it's a perfect place to calculate effective memory low and save it to use it for children cgroups.

This also eliminates a need to traverse the cgroup tree from bottom to top each time to check if parent's guarantee is not exceeded.

Setting/resetting effective memory.low is intentionally racy, but it's fine and shouldn't lead to any significant differences in actual memory distribution.

Quoted 6 lines

With this patch applied results are matching the expectations:

  A:    2146160640
  A/B:  1427795968
  A/C:  717705216
  A/D:  659456
  A/E:  0

Very cool results.

Below some comments on the implementation.

Quoted 19 lines

mm/memcontrol.c +19 -0

+static void memcg_update_low(struct mem_cgroup *memcg)
+{
+	unsigned long usage, low_usage, prev_low_usage;
+	struct mem_cgroup *parent;
+	long delta;
+
+	do {
+		parent = parent_mem_cgroup(memcg);
+		if (!parent || mem_cgroup_is_root(parent))
+			break;
+
+		if (!memcg->low && !atomic_long_read(&memcg->low_usage))
+			break;
+
+		usage = page_counter_read(&memcg->memory);
+		if (usage < memcg->low)
+			low_usage = usage;
+		else
+			low_usage = 0;

mm/memcontrol.c void unlock_page_memcg(struct page *page) · line 1703

Quoted 10 lines

mm/memcontrol.c +10 -0

+
+		prev_low_usage = atomic_long_xchg(&memcg->low_usage, low_usage);
+		delta = low_usage - prev_low_usage;
+		if (delta == 0)
+			break;
+
+		atomic_long_add(delta, &parent->s_low_usage);
+
+	} while ((memcg = parent));
+}

This code could use some comments ;)

Something that explains that we're tracking the combined usage of the children and what we're using that information for.

The conceptual descriptions you have in the changelog should be in the code somewher, to give a high level overview of how we're enforcing the low settings hierarchically.

mm/memcontrol.c static void drain_stock(struct memcg_stock_pcp *stock) · line 1761

Quoted 7 lines

mm/memcontrol.c +1 -0

@@ -1726,6 +1756,7 @@ static void drain_stock(struct memcg_stock_pcp *stock)
 		page_counter_uncharge(&old->memory, stock->nr_pages);
 		if (do_memsw_account())
 			page_counter_uncharge(&old->memsw, stock->nr_pages);
+		memcg_update_low(old);
 		css_put_many(&old->css, stock->nr_pages);
 		stock->nr_pages = 0;

The function is called every time the page counter changes and walks up the hierarchy exactly the same. That is a good sign that the low usage tracking should really be part of the page counter code itself.

I think you also have to call it when memory.low changes, as that may increase or decrease low usage just as much as when usage changes.

Quoted 41 lines · Roman Gushchin

Quoted 20 lines

This patch aims to address an issue in current memory.low semantics, which makes it hard to use it in a hierarchy, where some leaf memory cgroups are more valuable than others.

For example, there are memcgs A, A/B, A/C, A/D and A/E:

A A/memory.low = 2G, A/memory.current = 6G
//\\

BC  DE   B/memory.low = 3G  B/memory.usage = 2G
         C/memory.low = 1G  C/memory.usage = 2G
         D/memory.low = 0   D/memory.usage = 2G

E/memory.low = 10G E/memory.usage = 0

If we apply memory pressure, B, C and D are reclaimed at the same pace while A's usage exceeds 2G.
This is obviously wrong, as B's usage is fully below B's memory.low, and C has 1G of protection as well.
Also, A is pushed to the size, which is less than A's 2G memory.low, which is also wrong.

Quoted 7 lines

A simple bash script (provided below) can be used to reproduce the problem. Current results are:

  A:    1430097920
  A/B:  711929856
  A/C:  717426688
  A/D:  741376
  A/E:  0

Yes, this is a problem. And the behavior with your patch looks much preferable over the status quo.

Quoted 8 lines

To address the issue a concept of effective memory.low is introduced. Effective memory.low is always equal or less than original memory.low. In a case, when there is no memory.low overcommittment (and also for top-level cgroups), these two values are equal.
Otherwise it's a part of parent's effective memory.low, calculated as a cgroup's memory.low usage divided by sum of sibling's memory.low usages (under memory.low usage I mean the size of actually protected
memory: memory.current if memory.current < memory.low, 0 otherwise).

Quoted 8 lines

This hurts my brain.

Why is memory.current == memory.low (which should fully protect memory.current) a low usage of 0?

Why is memory.current > memory.low not a low usage of memory.low?

I.e. shouldn't this be low_usage = min(memory.current, memory.low)?

This is really the non-trivial part.

Let's look at an example:
memcg A (memory.current = 4G, memory.low = 2G)
memcg A/B (memory.current = 2G, memory.low = 2G)
memcg A/C (memory.current = 2G, memory.low = 1G)

If we'll calculate effective memory.low using your definition before any reclaim, we end up with the following:
A/B 2G * 2G / (2G + 1G) = 4/3G
A/C 2G * 1G / (2G + 1G) = 2/3G

Looks good, but both cgroups are below their effective limits. When memory pressure is applied, both are reclaimed at the same pace. While both B and C are getting smaller and smaller, their weights and effective low limits are getting closer and closer, but still below their usages. This ends up when both cgroups will have size of 1G, which is obviously wrong.

Fundamentally the problem is that memory.low doesn't define
the reclaim speed, just yes or no. So, if there are children cgroups, some of which are below their memory.low, and some above (as in the example), it's crucially important to reclaim unprotected memory first.

This is exactly what my code does: as soon as memory.current is larger than memory.low, we don't treat cgroup's memory as protected at all, so it doesn't affect effective limits of sibling cgroups.

Quoted 11 lines

Review static void drain_stock(struct memcg_stock_pcp *stock) · line 1761

Quoted 7 lines

(quoted diff) +1 -0

@@ -1726,6 +1756,7 @@ static void drain_stock(struct memcg_stock_pcp *stock)
 		page_counter_uncharge(&old->memory, stock->nr_pages);
 		if (do_memsw_account())
 			page_counter_uncharge(&old->memsw, stock->nr_pages);
+		memcg_update_low(old);
 		css_put_many(&old->css, stock->nr_pages);
 		stock->nr_pages = 0;

The function is called every time the page counter changes and walks up the hierarchy exactly the same. That is a good sign that the low usage tracking should really be part of the page counter code itself.

I thought about it, but the problem is that page counters are used for accounting swap, kmem, tcpmem (for v1), where low limit calculations are not applicable. I've no idea, how to add them nicely and without excessive overhead.
Also, good news are that it's possible to avoid any tracking until a user actually overcommits memory.low guarantees. I plan to implement this optimization in a separate patch.

Quoted 69 lines · Roman Gushchin

Quoted 41 lines · Roman Gushchin

Quoted 20 lines

This patch aims to address an issue in current memory.low semantics, which makes it hard to use it in a hierarchy, where some leaf memory cgroups are more valuable than others.

For example, there are memcgs A, A/B, A/C, A/D and A/E:

A A/memory.low = 2G, A/memory.current = 6G
//\\

BC  DE   B/memory.low = 3G  B/memory.usage = 2G
         C/memory.low = 1G  C/memory.usage = 2G
         D/memory.low = 0   D/memory.usage = 2G

E/memory.low = 10G E/memory.usage = 0

If we apply memory pressure, B, C and D are reclaimed at the same pace while A's usage exceeds 2G.
This is obviously wrong, as B's usage is fully below B's memory.low, and C has 1G of protection as well.
Also, A is pushed to the size, which is less than A's 2G memory.low, which is also wrong.

Quoted 7 lines

A simple bash script (provided below) can be used to reproduce the problem. Current results are:

  A:    1430097920
  A/B:  711929856
  A/C:  717426688
  A/D:  741376
  A/E:  0

Yes, this is a problem. And the behavior with your patch looks much preferable over the status quo.

Quoted 8 lines

To address the issue a concept of effective memory.low is introduced. Effective memory.low is always equal or less than original memory.low. In a case, when there is no memory.low overcommittment (and also for top-level cgroups), these two values are equal.
Otherwise it's a part of parent's effective memory.low, calculated as a cgroup's memory.low usage divided by sum of sibling's memory.low usages (under memory.low usage I mean the size of actually protected
memory: memory.current if memory.current < memory.low, 0 otherwise).

Quoted 8 lines

This hurts my brain.

Why is memory.current == memory.low (which should fully protect memory.current) a low usage of 0?

Why is memory.current > memory.low not a low usage of memory.low?

I.e. shouldn't this be low_usage = min(memory.current, memory.low)?

This is really the non-trivial part.

Let's look at an example:
memcg A (memory.current = 4G, memory.low = 2G)
memcg A/B (memory.current = 2G, memory.low = 2G)
memcg A/C (memory.current = 2G, memory.low = 1G)

If we'll calculate effective memory.low using your definition before any reclaim, we end up with the following:
A/B 2G * 2G / (2G + 1G) = 4/3G
A/C 2G * 1G / (2G + 1G) = 2/3G

Looks good, but both cgroups are below their effective limits. When memory pressure is applied, both are reclaimed at the same pace. While both B and C are getting smaller and smaller, their weights and effective low limits are getting closer and closer, but still below their usages. This ends up when both cgroups will have size of 1G, which is obviously wrong.

Quoted 8 lines

Fundamentally the problem is that memory.low doesn't define
the reclaim speed, just yes or no. So, if there are children cgroups, some of which are below their memory.low, and some above (as in the example), it's crucially important to reclaim unprotected memory first.

This is exactly what my code does: as soon as memory.current is larger than memory.low, we don't treat cgroup's memory as protected at all, so it doesn't affect effective limits of sibling cgroups.

Okay, that explanation makes sense to me. Once you're in excess, your memory is generally unprotected wrt your siblings until you're reigned in again.

It should still be usage <= low rather than usage < low, right? Since you're protected up to and including what that number says.

Quoted 12 lines

Quoted 11 lines

Review static void drain_stock(struct memcg_stock_pcp *stock) · line 1761

Quoted 7 lines

(quoted diff) +1 -0

@@ -1726,6 +1756,7 @@ static void drain_stock(struct memcg_stock_pcp *stock)
 		page_counter_uncharge(&old->memory, stock->nr_pages);
 		if (do_memsw_account())
 			page_counter_uncharge(&old->memsw, stock->nr_pages);
+		memcg_update_low(old);
 		css_put_many(&old->css, stock->nr_pages);
 		stock->nr_pages = 0;

The function is called every time the page counter changes and walks up the hierarchy exactly the same. That is a good sign that the low usage tracking should really be part of the page counter code itself.

Quoted 7 lines

I thought about it, but the problem is that page counters are used for accounting swap, kmem, tcpmem (for v1), where low limit calculations are not applicable. I've no idea, how to add them nicely and without excessive overhead.
Also, good news are that it's possible to avoid any tracking until a user actually overcommits memory.low guarantees. I plan to implement this optimization in a separate patch.

Hm, I'm not too worried about swap (not a sensitive path) or the other users (per-cpu batched). It just adds a branch. How about the below?

include/linux/page_counter.h +0 -0

index c15ab80ad32d..95bdbca86751 100644

Quoted 7 lines · Johannes Weiner

include/linux/page_counter.h +1 -0

@@ -9,8 +9,13 @@
 struct page_counter {
 	atomic_long_t count;
 	unsigned long limit;
+	unsigned long protected;
 	struct page_counter *parent;
 

Review line 18

Quoted 4 lines

(quoted diff) +4 -0

+	/* Hierarchical, proportional protection */
+	atomic_long_t protected_count;
+	atomic_long_t children_protected_count;
+

I followed your approach, but without introducing the new "protected" term. It looks cute in the usage tracking part, but a bit weird in mem_cgroup_low(), and it's not clear how to reuse it for memory.min. I think, we shouldn't introduce a new term without strict necessity.

Also, I moved the low field from memcg to page_counter, what made the code simpler and cleaner.

include/linux/memcontrol.h +1 -2

@@ -178,8 +178,7 @@ struct mem_cgroup {
 	struct page_counter kmem;
 	struct page_counter tcpmem;
 
-	/* Normal memory consumption range */
-	unsigned long low;
+	/* Upper bound of normal memory consumption range */
 	unsigned long high;
 
 	/* Range enforcement for interrupt charges */

include/linux/page_counter.h +0 -0

index c15ab80ad32d..e916595fb700 100644

include/linux/page_counter.h +33 -18

- * @root is exclusive; it is never low when looked at directly and isn't
- * checked when traversing the hierarchy.
+ * @root is exclusive; it is never low when looked at directly
  *
- * Excluding @root enables using memory.low to prioritize memory usage
- * between cgroups within a subtree of the hierarchy that is limited by
- * memory.high or memory.max.
+ * To provide a proper hierarchical behavior, effective memory.low value
+ * is used.
  *
- * For example, given cgroup A with children B and C:
+ * Effective memory.low is always equal or less than the original memory.low.
+ * If there is no memory.low overcommittment (which is always true for
+ * top-level memory cgroups), these two values are equal.
+ * Otherwise, it's a part of parent's effective memory.low,
+ * calculated as a cgroup's memory.low usage divided by sum of sibling's
+ * memory.low usages, where memory.low usage is the size of actually
+ * protected memory.
  *
- *    A
- *   / \
- *  B   C
+ *                                             low_usage
+ * elow = min( memory.low, parent->elow * ------------------ ),
+ *                                        siblings_low_usage
  *
- * and
+ *             | memory.current, if memory.current < memory.low
+ * low_usage = |
+	       | 0, otherwise.
  *
- *  1. A/memory.current > A/memory.high
- *  2. A/B/memory.current < A/B/memory.low
- *  3. A/C/memory.current >= A/C/memory.low
  *
- * As 'A' is high, i.e. triggers reclaim from 'A', and 'B' is low, we
- * should reclaim from 'C' until 'A' is no longer high or until we can
- * no longer reclaim from 'C'.  If 'A', i.e. @root, isn't excluded by
- * mem_cgroup_low when reclaming from 'A', then 'B' won't be considered
- * low and we will reclaim indiscriminately from both 'B' and 'C'.
+ * Such definition of the effective memory.low provides the expected
+ * hierarchical behavior: parent's memory.low value is limiting
+ * children, unprotected memory is reclaimed first and cgroups,
+ * which are not using their guarantee do not affect actual memory
+ * distribution.
+ *
+ * For example, if there are memcgs A, A/B, A/C, A/D and A/E:
+ *
+ *     A      A/memory.low = 2G, A/memory.current = 6G
+ *    //\\
+ *   BC  DE   B/memory.low = 3G  B/memory.current = 2G
+ *            C/memory.low = 1G  C/memory.current = 2G
+ *            D/memory.low = 0   D/memory.current = 2G
+ *            E/memory.low = 10G E/memory.current = 0
+ *
+ * and the memory pressure is applied, the following memory distribution
+ * is expected (approximately):

include/linux/page_counter.h +9 -1

+ * effective memory.low values. But as we do call mem_cgroup_low()
+ * path for each memory cgroup top-down from the reclaim,
+ * it's possible to optimize this part, and save calculated elow
+ * for next usage. This part is intentionally racy, but it's ok,
+ * as memory.low is a best-effort mechanism.
  */
 bool mem_cgroup_low(struct mem_cgroup *root, struct mem_cgroup *memcg)
 {
+	unsigned long usage, low_usage, siblings_low_usage;
+	unsigned long elow, parent_elow;
+	struct mem_cgroup *parent;
+
 	if (mem_cgroup_disabled())
 		return false;
 

@@ -5650,12 +5683,31 @@ bool mem_cgroup_low(struct mem_cgroup *root, struct mem_cgroup *memcg)
 	if (memcg == root)
 		return false;
 
-	for (; memcg != root; memcg = parent_mem_cgroup(memcg)) {

include/linux/page_counter.h +16 -0

@@ -13,6 +13,34 @@
 #include <linux/bug.h>
 #include <asm/page.h>
 
+static void propagate_low_usage(struct page_counter *c, unsigned long usage)
+{
+	unsigned long low_usage, old;
+	long delta;
+
+	if (!c->parent)
+		return;
+
+	if (!c->low && !atomic_long_read(&c->low_usage))
+		return;
+
+	if (usage <= c->low)
+		low_usage = usage;
+	else
+		low_usage = 0;
+

Roman's previously posted memory.low,min patches add per memcg effective low limit to detect overcommitment of parental limits. But if we flip low,min reclaim to bail if usage<{low,min} at any level, then we don't need an effective low limit, which makes the code simpler. When parent limits are overcommited memory.min will oom kill, which is more drastic but makes the memory.low a simpler concept. If memcg a/b wants oom kill before reclaim, then give it to them. It seems a bit strange for a/b/memory.low's behaviour to depend on a/c/memory.low (i.e. a/b.low is strong unless a/b.low+a/c.low exceed a.low).

I think there might be a simpler way (ableit it doesn't yet include Documentation):
- memcg: fix memory.low
- memcg: add memory.min

 3 files changed, 75 insertions(+), 6 deletions(-)

The idea of this alternate approach is for memory.low,min to avoid reclaim if any portion of under-consideration memcg ancestry is under respective limit.

Hi, Greg!

Quoted 3 lines · Greg Thelen

Roman's previously posted memory.low,min patches add per memcg effective low limit to detect overcommitment of parental limits. But if we flip low,min reclaim to bail if usage<{low,min} at any level, then we don't

Quoted 1 line

an effective low limit, which makes the code simpler. When parent

Quoted 1 line

are overcommited memory.min will oom kill, which is more drastic but

Quoted 2 lines

the memory.low a simpler concept. If memcg a/b wants oom kill before reclaim, then give it to them. It seems a bit strange for

Quoted 2 lines

behaviour to depend on a/c/memory.low (i.e. a/b.low is strong unless a/b.low+a/c.low exceed a.low).

It's actually not strange: a/b and a/c are sharing a common resource:
a/memory.low.

Exactly as a/b/memory.max and a/c/memory.max are sharing a/memory.max. If there are sibling cgroups which are consuming memory, a cgroup can't exceed parent's memory.max, even if its memory.max is grater.

Quoted 8 lines

I think there might be a simpler way (ableit it doesn't yet include Documentation):
- memcg: fix memory.low
- memcg: add memory.min

 3 files changed, 75 insertions(+), 6 deletions(-)

The idea of this alternate approach is for memory.low,min to avoid

Quoted 2 lines

if any portion of under-consideration memcg ancestry is under respective limit.

This approach has a significant downside: it breaks hierarchical

for memory.low/min. There are two important outcomes:

1) Any leaf's memory.low/min value is respected, even if parent's value is lower or even 0. It's not possible anymore to limit the amount

protected memory for a sub-tree. This is especially bad in case of delegation.

2) If a cgroup has an ancestor with the usage under its memory.low/min, it becomes protection, even if its memory.low/min is 0. So it

impossible to have unprotected cgroups in protected sub-tree.

Quoted 13 lines · Roman Gushchin

Hi, Greg!

Quoted 9 lines · Greg Thelen

Roman's previously posted memory.low,min patches add per memcg effective low limit to detect overcommitment of parental limits. But if we flip low,min reclaim to bail if usage<{low,min} at any level, then we don't need an effective low limit, which makes the code simpler. When parent limits are overcommited memory.min will oom kill, which is more drastic but makes the memory.low a simpler concept. If memcg a/b wants oom kill before reclaim, then give it to them. It seems a bit strange for a/b/memory.low's behaviour to depend on a/c/memory.low (i.e. a/b.low is strong unless a/b.low+a/c.low exceed a.low).

Quoted 15 lines

It's actually not strange: a/b and a/c are sharing a common resource:
a/memory.low.

Exactly as a/b/memory.max and a/c/memory.max are sharing a/memory.max. If there are sibling cgroups which are consuming memory, a cgroup can't exceed parent's memory.max, even if its memory.max is grater.

Quoted 8 lines

I think there might be a simpler way (ableit it doesn't yet include Documentation):
- memcg: fix memory.low
- memcg: add memory.min

 3 files changed, 75 insertions(+), 6 deletions(-)

The idea of this alternate approach is for memory.low,min to avoid

reclaim

Quoted 4 lines

Quoted 2 lines

if any portion of under-consideration memcg ancestry is under respective limit.

This approach has a significant downside: it breaks hierarchical

constraints

Quoted 4 lines

for memory.low/min. There are two important outcomes:

1) Any leaf's memory.low/min value is respected, even if parent's value is lower or even 0. It's not possible anymore to limit the amount

of

Quoted 2 lines

protected memory for a sub-tree. This is especially bad in case of delegation.

As someone who has been using something like memory.min for a while, I have cases where it needs to be a strong protection. Such jobs prefer oom kill to reclaim. These jobs know they need X MB of memory. But I guess it's on me to avoid configuring machines which overcommit memory.min at such cgroup levels all the way to the root.

Quoted 3 lines

2) If a cgroup has an ancestor with the usage under its memory.low/min, it becomes protection, even if its memory.low/min is 0. So it

becomes

Quoted 1 line

impossible to have unprotected cgroups in protected sub-tree.

Fair point.

One use case is where a non trivial job which has several memory accounting subcontainers. Is there a way to only set memory.low at the top and have the offer protection to the job?
The case I'm thinking of is:
% cd /cgroup
% echo +memory > cgroup.subtree_control
% mkdir top
% echo +memory > top/cgroup.subtree_control
% mkdir top/part1 top/part2
% echo 1GB > top/memory.min
% (echo $BASHPID > top/part1/cgroup.procs && part1)
% (echo $BASHPID > top/part2/cgroup.procs && part2)

Empirically it's been measured that the entire workload (/top) needs 1GB to perform well. But we don't care how the memory is distributed between part1,part2. Is the strategy for that to set /top, /top/part1.min, and /top/part2.min to 1GB?

What do you think about exposing emin and elow to user space? I think that would reduce admin/user confusion in situations where memory.min is internally discounted.

(tangent) Delegation in v2 isn't something I've been able to fully internalize yet.
The "no interior processes" rule challenges my notion of subdelegation. My current model is where a system controller creates a container C with C.min and then starts client manager process M in C. Then M can choose to further divide C's resources (e.g. C/S). This doesn't seem possible because v2 doesn't allow for interior processes. So the system manager would need to create C, set C.low, create C/sub_manager, create C/sub_resources, set C/sub_manager.low, set C/sub_resources.low, then start M in C/sub_manager. Then sub_manager can create and manage C/sub_resources/S.