Skip to contentSkip to conversation

memcg: robust enforcement of memory.high

PATCH Applied Shakeel Butt 3 participants 11 messages Feb 11, 2022 – Feb 16, 2022

Subsystems (4)
lore
Share & tools
lore
#0 lore raw

v2 cover memcg: robust enforcement of memory.high

Due to the semantics of memory.high enforcement i.e. throttle the workload without oom-kill, we are trying to use it for right sizing the workloads in our production environment. However we observed the mechanism fails for some specific applications which does big chunck of allocations in a single syscall. The reason behind this failure is due to the limitation of the memory.high enforcement's current implementation. This patch series solves this issue by enforcing the memory.high synchronously if the current process has accumulated a large amount of high overcharge.

Changes since v1:
- Based on Roman's comment simply the sync enforcement and only target the extreme cases.

Shakeel Butt (4):
memcg: refactor mem_cgroup_oom
memcg: unify force charging conditions
selftests: memcg: test high limit for single entry allocation
memcg: synchronously enforce memory.high for large overcharges

 mm/memcontrol.c                               | 66 +++++++---------
 tools/testing/selftests/cgroup/cgroup_util.c  | 15 +++-
 tools/testing/selftests/cgroup/cgroup_util.h  |  1 +
 .../selftests/cgroup/test_memcontrol.c        | 78 +++++++++++++++++++
 4 files changed, 120 insertions(+), 40 deletions(-)
Signature · 1 line
2.35.1.265.g69c8d7142f-goog
#1 lore raw

PATCH v2 1/4 memcg: refactor mem_cgroup_oom

The function mem_cgroup_oom returns enum which has four possible values but the caller does not care about such values and only cares if the return value is OOM_SUCCESS or not. So, remove the enum altogether and make mem_cgroup_oom returns a simple bool.

---
Changes since v1:
- Added comment for mem_cgroup_oom as suggested by Roman

 mm/memcontrol.c | 44 +++++++++++++++++---------------------------
 1 file changed, 17 insertions(+), 27 deletions(-)
mm/memcontrol.c +17 -27
index a0e9d9f12cf5..f12e489ba9b8 100644
@@ -1795,20 +1795,16 @@ static void memcg_oom_recover(struct mem_cgroup *memcg)
 		__wake_up(&memcg_oom_waitq, TASK_NORMAL, 0, memcg);
 }
 
-enum oom_status {
-	OOM_SUCCESS,
-	OOM_FAILED,
-	OOM_ASYNC,
-	OOM_SKIPPED
-};
-
-static enum oom_status mem_cgroup_oom(struct mem_cgroup *memcg, gfp_t mask, int order)
+/*
+ * Returns true if successfully killed one or more processes. Though in some
+ * corner cases it can return true even without killing any process.
+ */
+static bool mem_cgroup_oom(struct mem_cgroup *memcg, gfp_t mask, int order)
 {
-	enum oom_status ret;
-	bool locked;
+	bool locked, ret;
 
 	if (order > PAGE_ALLOC_COSTLY_ORDER)
-		return OOM_SKIPPED;
+		return false;
 
 	memcg_memory_event(memcg, MEMCG_OOM);
 
@@ -1831,14 +1827,13 @@ static enum oom_status mem_cgroup_oom(struct mem_cgroup *memcg, gfp_t mask, int
 	 * victim and then we have to bail out from the charge path.
 	 */
 	if (memcg->oom_kill_disable) {
-		if (!current->in_user_fault)
-			return OOM_SKIPPED;
-		css_get(&memcg->css);
-		current->memcg_in_oom = memcg;
-		current->memcg_oom_gfp_mask = mask;
-		current->memcg_oom_order = order;
-
-		return OOM_ASYNC;
+		if (current->in_user_fault) {
+			css_get(&memcg->css);
+			current->memcg_in_oom = memcg;
+			current->memcg_oom_gfp_mask = mask;
+			current->memcg_oom_order = order;
+		}
+		return false;
 	}
 
 	mem_cgroup_mark_under_oom(memcg);
@@ -1849,10 +1844,7 @@ static enum oom_status mem_cgroup_oom(struct mem_cgroup *memcg, gfp_t mask, int
 		mem_cgroup_oom_notify(memcg);
 
 	mem_cgroup_unmark_under_oom(memcg);
-	if (mem_cgroup_out_of_memory(memcg, mask, order))
-		ret = OOM_SUCCESS;
-	else
-		ret = OOM_FAILED;
+	ret = mem_cgroup_out_of_memory(memcg, mask, order);
 
 	if (locked)
 		mem_cgroup_oom_unlock(memcg);
@@ -2545,7 +2537,6 @@ static int try_charge_memcg(struct mem_cgroup *memcg, gfp_t gfp_mask,
 	int nr_retries = MAX_RECLAIM_RETRIES;
 	struct mem_cgroup *mem_over_limit;
 	struct page_counter *counter;
-	enum oom_status oom_status;
 	unsigned long nr_reclaimed;
 	bool passed_oom = false;
 	bool may_swap = true;
@@ -2648,9 +2639,8 @@ static int try_charge_memcg(struct mem_cgroup *memcg, gfp_t gfp_mask,
 	 * a forward progress or bypass the charge if the oom killer
 	 * couldn't make any progress.
 	 */
-	oom_status = mem_cgroup_oom(mem_over_limit, gfp_mask,
-		       get_order(nr_pages * PAGE_SIZE));
-	if (oom_status == OOM_SUCCESS) {
+	if (mem_cgroup_oom(mem_over_limit, gfp_mask,
+			   get_order(nr_pages * PAGE_SIZE))) {
 		passed_oom = true;
 		nr_retries = MAX_RECLAIM_RETRIES;
 		goto retry;
Signature · 1 line
2.35.1.265.g69c8d7142f-goog
#2 lore raw

PATCH v2 2/4 memcg: unify force charging conditions

In reply to Shakeel Butt

Currently the kernel force charges the allocations which have __GFP_HIGH flag without triggering the memory reclaim. __GFP_HIGH indicates that the caller is high priority and since commit 869712fd3de5 ("mm:
memcontrol: fix network errors from failing __GFP_ATOMIC charges") the kernel lets such allocations do force charging. Please note that __GFP_ATOMIC has been replaced by __GFP_HIGH.

__GFP_HIGH does not tell if the caller can block or can trigger reclaim. There are separate checks to determine that. So, there is no need to skip reclaiming for __GFP_HIGH allocations. So, handle __GFP_HIGH together with __GFP_NOFAIL which also does force charging.

Please note that this is a noop change as there are no __GFP_HIGH allocators in the kernel which also have __GFP_ACCOUNT (or SLAB_ACCOUNT) and does not allow reclaim for now.

---
Changes since v1:
- None

 mm/memcontrol.c | 17 +++++++----------
 1 file changed, 7 insertions(+), 10 deletions(-)
mm/memcontrol.c +7 -10
index f12e489ba9b8..292b0b99a2c7 100644
@@ -2564,15 +2564,6 @@ static int try_charge_memcg(struct mem_cgroup *memcg, gfp_t gfp_mask,
 		goto retry;
 	}
 
-	/*
-	 * Memcg doesn't have a dedicated reserve for atomic
-	 * allocations. But like the global atomic pool, we need to
-	 * put the burden of reclaim on regular allocation requests
-	 * and let these go through as privileged allocations.
-	 */
-	if (gfp_mask & __GFP_HIGH)
-		goto force;
-
 	/*
 	 * Prevent unbounded recursion when reclaim operations need to
 	 * allocate memory. This might exceed the limits temporarily,
@@ -2646,7 +2637,13 @@ static int try_charge_memcg(struct mem_cgroup *memcg, gfp_t gfp_mask,
 		goto retry;
 	}
 nomem:
-	if (!(gfp_mask & __GFP_NOFAIL))
+	/*
+	 * Memcg doesn't have a dedicated reserve for atomic
+	 * allocations. But like the global atomic pool, we need to
+	 * put the burden of reclaim on regular allocation requests
+	 * and let these go through as privileged allocations.
+	 */
+	if (!(gfp_mask & (__GFP_NOFAIL | __GFP_HIGH)))
 		return -ENOMEM;
 force:
 	/*
Signature · 1 line
2.35.1.265.g69c8d7142f-goog
#3 lore raw

PATCH v2 3/4 selftests: memcg: test high limit for single entry allocation

In reply to Shakeel Butt

Test the enforcement of memory.high limit for large amount of memory allocation within a single kernel entry. There are valid use-cases where the application can trigger large amount of memory allocation within a single syscall e.g. mlock() or mmap(MAP_POPULATE). Make sure memory.high limit enforcement works for such use-cases.

---
Changes since v1:
- None

 tools/testing/selftests/cgroup/cgroup_util.c  | 15 +++-
 tools/testing/selftests/cgroup/cgroup_util.h  |  1 +
 .../selftests/cgroup/test_memcontrol.c        | 78 +++++++++++++++++++
 3 files changed, 91 insertions(+), 3 deletions(-)
tools/testing/selftests/cgroup/cgroup_util.c +12 -3
index 0cf7e90c0052..dbaa7aabbb4a 100644
@@ -583,7 +583,7 @@ int clone_into_cgroup_run_wait(const char *cgroup)
 	return 0;
 }
 
-int cg_prepare_for_wait(const char *cgroup)
+static int __prepare_for_wait(const char *cgroup, const char *filename)
 {
 	int fd, ret = -1;
 
@@ -591,8 +591,7 @@ int cg_prepare_for_wait(const char *cgroup)
 	if (fd == -1)
 		return fd;
 
-	ret = inotify_add_watch(fd, cg_control(cgroup, "cgroup.events"),
-				IN_MODIFY);
+	ret = inotify_add_watch(fd, cg_control(cgroup, filename), IN_MODIFY);
 	if (ret == -1) {
 		close(fd);
 		fd = -1;
@@ -601,6 +600,16 @@ int cg_prepare_for_wait(const char *cgroup)
 	return fd;
 }
 
+int cg_prepare_for_wait(const char *cgroup)
+{
+	return __prepare_for_wait(cgroup, "cgroup.events");
+}
+
+int memcg_prepare_for_wait(const char *cgroup)
+{
+	return __prepare_for_wait(cgroup, "memory.events");
+}
+
 int cg_wait_for(int fd)
 {
 	int ret = -1;
tools/testing/selftests/cgroup/cgroup_util.h +1 -0
index 4f66d10626d2..628738532ac9 100644
@@ -55,4 +55,5 @@ extern int clone_reap(pid_t pid, int options);
 extern int clone_into_cgroup_run_wait(const char *cgroup);
 extern int dirfd_open_opath(const char *dir);
 extern int cg_prepare_for_wait(const char *cgroup);
+extern int memcg_prepare_for_wait(const char *cgroup);
 extern int cg_wait_for(int fd);
tools/testing/selftests/cgroup/test_memcontrol.c +78 -0

96 lines not on this page · raw message

Signature · 1 line
2.35.1.265.g69c8d7142f-goog
Reviewed-by on v2 3/4
#4 lore raw

Re: [PATCH v2 3/4] selftests: memcg: test high limit for single entry allocation

Quoted 10 lines · Shakeel Butt

Test the enforcement of memory.high limit for large amount of memory allocation within a single kernel entry. There are valid use-cases where the application can trigger large amount of memory allocation within a single syscall e.g. mlock() or mmap(MAP_POPULATE). Make sure memory.high limit enforcement works for such use-cases.

---
Changes since v1:
- None

Thanks!

#5 lore raw

PATCH v2 4/4 memcg: synchronously enforce memory.high for large overcharges

In reply to Shakeel Butt

The high limit is used to throttle the workload without invoking the oom-killer. Recently we tried to use the high limit to right size our internal workloads. More specifically dynamically adjusting the limits of the workload without letting the workload get oom-killed. However due to the limitation of the implementation of high limit enforcement, we observed the mechanism fails for some real workloads.

The high limit is enforced on return-to-userspace i.e. the kernel let the usage goes over the limit and when the execution returns to userspace, the high reclaim is triggered and the process can get throttled as well. However this mechanism fails for workloads which do large allocations in a single kernel entry e.g. applications that mlock() a large chunk of memory in a single syscall. Such applications bypass the high limit and can trigger the oom-killer.

To make high limit enforcement more robust, this patch makes the limit enforcement synchronous only if the accumulated overcharge becomes larger than MEMCG_CHARGE_BATCH. So, most of the allocations would still be throttled on the return-to-userspace path but only the extreme allocations which accumulates large amount of overcharge without returning to the userspace will be throttled synchronously. The value MEMCG_CHARGE_BATCH is a bit arbitrary but most of other places in the memcg codebase uses this constant therefore for now uses the same one.

---
Changes since v1:
- Based on Roman's comment simply the sync enforcement and only target the extreme cases.

 mm/memcontrol.c | 5 +++++
 1 file changed, 5 insertions(+)
mm/memcontrol.c +5 -0
index 292b0b99a2c7..0da4be4798e7 100644
@@ -2703,6 +2703,11 @@ static int try_charge_memcg(struct mem_cgroup *memcg, gfp_t gfp_mask,
 		}
 	} while ((memcg = parent_mem_cgroup(memcg)));
 
+	if (current->memcg_nr_pages_over_high > MEMCG_CHARGE_BATCH &&
+	    !(current->flags & PF_MEMALLOC) &&
+	    gfpflags_allow_blocking(gfp_mask)) {
+		mem_cgroup_handle_over_high();
+	}
 	return 0;
 }
 
Signature · 1 line
2.35.1.265.g69c8d7142f-goog
#6 lore raw

Re: [PATCH v2 4/4] memcg: synchronously enforce memory.high for large overcharges

Quoted 15 lines · Shakeel Butt

The high limit is used to throttle the workload without invoking the oom-killer. Recently we tried to use the high limit to right size our internal workloads. More specifically dynamically adjusting the limits of the workload without letting the workload get oom-killed. However due to the limitation of the implementation of high limit enforcement, we observed the mechanism fails for some real workloads.

The high limit is enforced on return-to-userspace i.e. the kernel let the usage goes over the limit and when the execution returns to userspace, the high reclaim is triggered and the process can get throttled as well. However this mechanism fails for workloads which do large allocations in a single kernel entry e.g. applications that mlock() a large chunk of memory in a single syscall. Such applications bypass the high limit and can trigger the oom-killer.

Quoted 8 lines

To make high limit enforcement more robust, this patch makes the limit enforcement synchronous only if the accumulated overcharge becomes larger than MEMCG_CHARGE_BATCH. So, most of the allocations would still be throttled on the return-to-userspace path but only the extreme allocations which accumulates large amount of overcharge without returning to the userspace will be throttled synchronously. The value MEMCG_CHARGE_BATCH is a bit arbitrary but most of other places in the memcg codebase uses this constant therefore for now uses the same one.

Note that mem_cgroup_handle_over_high() has its own allocator throttling grace period, where it bails out if the penalty to apply is less than 10ms. The reclaim will still happen, though. So throttling might not happen even for roughly MEMCG_CHARGE_BATCH-sized allocations, depending on the overall size of the cgroup and its protection.

Quoted 14 lines

---
Changes since v1:
- Based on Roman's comment simply the sync enforcement and only target the extreme cases.

mm/memcontrol.c | 5 +++++

1 file changed, 5 insertions(+)
mm/memcontrol.c +0 -0
index 292b0b99a2c7..0da4be4798e7 100644

mm/memcontrol.c static int try_charge_memcg(struct mem_cgroup *memcg, gfp_t gfp_mask, · line 2709

Quoted 7 lines
mm/memcontrol.c +4 -0
@@ -2703,6 +2703,11 @@ static int try_charge_memcg(struct mem_cgroup *memcg, gfp_t gfp_mask,
 		}
 	} while ((memcg = parent_mem_cgroup(memcg)));
 
+	if (current->memcg_nr_pages_over_high > MEMCG_CHARGE_BATCH &&
+	    !(current->flags & PF_MEMALLOC) &&
+	    gfpflags_allow_blocking(gfp_mask)) {
+		mem_cgroup_handle_over_high();

Thanks, I was going to comment on v1 that I prefer to keep the implementation of mem_cgroup_handle_over_high if possible since we know that the mechanism has been safe in production over the past few years.

One question I have is about throttling. It looks like this new mem_cgroup_handle_over_high callsite may mean that throttling is invoked more than once on a misbehaving workload that's failing to reclaim since the throttling could be invoked both here and in return to userspace, right? That might not be a problem, but we should think about the implications of that, especially in relation to MEMCG_MAX_HIGH_DELAY_JIFFIES.

Maybe we should record if throttling happened previously and avoid doing it again for this entry into kernelspace? Not certain that's the right answer, but we should think about what the new semantics should be.

Quoted 7 lines
mm/memcontrol.c +1 -0
+	}
 	return 0;
 }
 
Signature · 1 line
2.35.1.265.g69c8d7142f-goog
#7 lore raw

Re: [PATCH v2 4/4] memcg: synchronously enforce memory.high for large overcharges

Quoted 1 line · Chris Down

[...]

Quoted 9 lines
Quoted 8 lines

To make high limit enforcement more robust, this patch makes the limit enforcement synchronous only if the accumulated overcharge becomes larger than MEMCG_CHARGE_BATCH. So, most of the allocations would still be throttled on the return-to-userspace path but only the extreme allocations which accumulates large amount of overcharge without returning to the userspace will be throttled synchronously. The value MEMCG_CHARGE_BATCH is a bit arbitrary but most of other places in the memcg codebase uses this constant therefore for now uses the same one.

Quoted 6 lines

Note that mem_cgroup_handle_over_high() has its own allocator throttling grace period, where it bails out if the penalty to apply is less than 10ms. The reclaim will still happen, though. So throttling might not happen even for roughly MEMCG_CHARGE_BATCH-sized allocations, depending on the overall size of the cgroup and its protection.

Here by throttling, I meant both reclaim and schedule_timeout_killable(). I don't want to say low level details which might change in future.

[...]

Quoted 5 lines

Thanks, I was going to comment on v1 that I prefer to keep the implementation of mem_cgroup_handle_over_high if possible since we know that the mechanism has been safe in production over the past few years.

Quoted 7 lines

One question I have is about throttling. It looks like this new mem_cgroup_handle_over_high callsite may mean that throttling is invoked more than once on a misbehaving workload that's failing to reclaim since the throttling could be invoked both here and in return to userspace, right? That might not be a problem, but we should think about the implications of that, especially in relation to MEMCG_MAX_HIGH_DELAY_JIFFIES.

Please note that mem_cgroup_handle_over_high() clears memcg_nr_pages_over_high and if on the return-to-userspace path mem_cgroup_handle_over_high() finds that memcg_nr_pages_over_high is non-zero, then it means the task has further accumulated the charges over high limit after a possibly synchronous memcg_nr_pages_over_high() call.

Quoted 3 lines

Maybe we should record if throttling happened previously and avoid doing it again for this entry into kernelspace? Not certain that's the right answer, but we should think about what the new semantics should be.

For now, I will keep this as is and will add a comment in the code and a mention in the commit message about it. I will wait for others to comment before sending the next version and thanks for taking a look.

Acked-by
#8 lore raw

Re: [PATCH v2 4/4] memcg: synchronously enforce memory.high for large overcharges

Quoted 12 lines · Shakeel Butt
Quoted 11 lines

Thanks, I was going to comment on v1 that I prefer to keep the implementation of mem_cgroup_handle_over_high if possible since we know that the mechanism has been safe in production over the past few years.

One question I have is about throttling. It looks like this new mem_cgroup_handle_over_high callsite may mean that throttling is invoked more than once on a misbehaving workload that's failing to reclaim since the throttling could be invoked both here and in return to userspace, right? That might not be a problem, but we should think about the implications of that, especially in relation to MEMCG_MAX_HIGH_DELAY_JIFFIES.

Quoted 6 lines

Please note that mem_cgroup_handle_over_high() clears memcg_nr_pages_over_high and if on the return-to-userspace path mem_cgroup_handle_over_high() finds that memcg_nr_pages_over_high is non-zero, then it means the task has further accumulated the charges over high limit after a possibly synchronous memcg_nr_pages_over_high() call.

Oh sure, my point was only that MEMCG_MAX_HIGH_DELAY_JIFFIES was to more reliably ensure we are returning to userspace at some point in the near future to allow the task to have another chance at good behaviour instead of being immediately whacked with whatever is monitoring PSI -- for example, in the case where we have a daemon which is monitoring its own PSI contributions and will make a proactive attempt to free structures in userspace.

That said, the throttling here still isn't unbounded, and it's not likely that anyone doing such large allocations after already exceeding memory.high is being a good citizen, so I think the patch makes sense as long as the change is understood and documented internally.

Thanks!

#9 lore raw

Re: [PATCH v2 4/4] memcg: synchronously enforce memory.high for large overcharges

In reply to Shakeel Butt
Quoted 25 lines · Shakeel Butt

The high limit is used to throttle the workload without invoking the oom-killer. Recently we tried to use the high limit to right size our internal workloads. More specifically dynamically adjusting the limits of the workload without letting the workload get oom-killed. However due to the limitation of the implementation of high limit enforcement, we observed the mechanism fails for some real workloads.

The high limit is enforced on return-to-userspace i.e. the kernel let the usage goes over the limit and when the execution returns to userspace, the high reclaim is triggered and the process can get throttled as well. However this mechanism fails for workloads which do large allocations in a single kernel entry e.g. applications that mlock() a large chunk of memory in a single syscall. Such applications bypass the high limit and can trigger the oom-killer.

To make high limit enforcement more robust, this patch makes the limit enforcement synchronous only if the accumulated overcharge becomes larger than MEMCG_CHARGE_BATCH. So, most of the allocations would still be throttled on the return-to-userspace path but only the extreme allocations which accumulates large amount of overcharge without returning to the userspace will be throttled synchronously. The value MEMCG_CHARGE_BATCH is a bit arbitrary but most of other places in the memcg codebase uses this constant therefore for now uses the same one.

Quoted 1 line

Any comments or concerns on this patch? Otherwise I would ask Andrew to add this series into the mm tree.

Quoted 27 lines

---
Changes since v1:
- Based on Roman's comment simply the sync enforcement and only target the extreme cases.

 mm/memcontrol.c | 5 +++++
 1 file changed, 5 insertions(+)
mm/memcontrol.c +5 -1
index 292b0b99a2c7..0da4be4798e7 100644
@@ -2703,6 +2703,11 @@ static int try_charge_memcg(struct mem_cgroup *memcg, gfp_t gfp_mask,
                }
        } while ((memcg = parent_mem_cgroup(memcg)));
 
+       if (current->memcg_nr_pages_over_high > MEMCG_CHARGE_BATCH &&
+           !(current->flags & PF_MEMALLOC) &&
+           gfpflags_allow_blocking(gfp_mask)) {
+               mem_cgroup_handle_over_high();
+       }
        return 0;
 }
 
--

2.35.1.265.g69c8d7142f-goog

Reviewed-by on v2 4/4
#10 lore raw

Re: [PATCH v2 4/4] memcg: synchronously enforce memory.high for large overcharges

In reply to Shakeel Butt
Quoted 24 lines · Shakeel Butt

The high limit is used to throttle the workload without invoking the oom-killer. Recently we tried to use the high limit to right size our internal workloads. More specifically dynamically adjusting the limits of the workload without letting the workload get oom-killed. However due to the limitation of the implementation of high limit enforcement, we observed the mechanism fails for some real workloads.

The high limit is enforced on return-to-userspace i.e. the kernel let the usage goes over the limit and when the execution returns to userspace, the high reclaim is triggered and the process can get throttled as well. However this mechanism fails for workloads which do large allocations in a single kernel entry e.g. applications that mlock() a large chunk of memory in a single syscall. Such applications bypass the high limit and can trigger the oom-killer.

To make high limit enforcement more robust, this patch makes the limit enforcement synchronous only if the accumulated overcharge becomes larger than MEMCG_CHARGE_BATCH. So, most of the allocations would still be throttled on the return-to-userspace path but only the extreme allocations which accumulates large amount of overcharge without returning to the userspace will be throttled synchronously. The value MEMCG_CHARGE_BATCH is a bit arbitrary but most of other places in the memcg codebase uses this constant therefore for now uses the same one.

Quoted 5 lines

---
Changes since v1:
- Based on Roman's comment simply the sync enforcement and only target the extreme cases.

This version indeed looks more safe to me.

Thanks!