Skip to contentSkip to conversation

mmapstress06: Rate limit page-dirtying loop to prevent MemCG OOM-kills

PATCH In review Wei Gao via ltp 3 participants 19 messages Aug 25 – Sep 4 Library code

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 mmapstress06: Rate limit page-dirtying loop to prevent MemCG OOM-kills

Reduce the throttling interval from 2 MB to 64 KB (usleep for 200us every 64 KB) in the page-dirtying loop. This smooths out allocation spikes and gives asynchronous swap writeback time to catch up under slower VM storage environments, avoiding unexpected MemCG OOM-kills.

---
 testcases/kernel/mem/mmapstress/mmapstress06.c | 8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)
testcases/kernel/mem/mmapstress/mmapstress06.c +6 -2
index 4e6023399..2813ef185 100644
@@ -57,10 +57,14 @@ static void run_test(void)
 
 		tst_res(TINFO, "Dirtying %zu bytes in child", map_size);
 
+		/*
+		 * Yield periodically to space out allocation spikes and allow
+		 * asynchronous swap writeback to catch up on slower VM disks.
+		 */
 		for (size_t i = 0; i < map_size; i += page_size) {
 			mmapaddr[i] = 'a';
-			if ((i % (2 * 1024 * 1024)) == 0)
-				usleep(1000);
+			if ((i % (64 * 1024)) == 0)
+				usleep(200);
 		}
 
 		SAFE_CG_SCANF(cg_child, "memory.swap.current", "%lu", &cg_swap_after);
Signature · 5 lines
2.55.0


-- 
Mailing list info: https://lists.linux.it/listinfo/ltp
#1 lore raw

Hi Wei,

testcases/kernel/mem/mmapstress/mmapstress06.c static void run_test(void) · line 67

Quoted 12 lines · Wei Gao

mmapstress06: Rate limit page-dirtying loop to prevent MemCG OOM-kills

testcases/kernel/mem/mmapstress/mmapstress06.c +6 -2
+		/*
+		 * Yield periodically to space out allocation spikes and allow
+		 * asynchronous swap writeback to catch up on slower VM disks.
+		 */
 		for (size_t i = 0; i < map_size; i += page_size) {
 			mmapaddr[i] = 'a';
-			if ((i % (2 * 1024 * 1024)) == 0)
-				usleep(1000);
+			if ((i % (64 * 1024)) == 0)
+				usleep(200);

Could this avoid using a fixed sleep to synchronize with asynchronous swap writeback? The 7.2 cgroup documentation says memory.max invokes the cgroup OOM killer when reclaim cannot reduce usage, while the delay here is unrelated to actual reclaim progress. A swap device slower than the chosen dirtying rate can therefore still OOM-kill the child and make the test fail for environmental timing reasons. Please use reclaim throttling or poll observable progress with backoff instead of tuning a fixed delay.

Verdict - Needs revision

---
Note:

The agent can sometimes produce false positives although often its findings are genuine. If you find issues with the review, please comment this email or ignore the suggestions.

Regards,
LTP AI Reviewer

Signature · 1 line
Mailing list info: https://lists.linux.it/listinfo/ltp
#2 lore raw
In reply to Wei Gao via ltp

Hi Wei,

Quoted 4 lines

Reduce the throttling interval from 2 MB to 64 KB (usleep for 200us every 64 KB) in the page-dirtying loop. This smooths out allocation spikes and gives asynchronous swap writeback time to catch up under slower VM storage environments, avoiding unexpected MemCG OOM-kills.

Unfortunately this is not a solution but a workaround, we need to make sure that we are able to see any change to the swap memory while allocating dirty pages close to the memory.max. This is probably a fix for a specific target rather than complete fix.

We need to find an another way to ensure that OOM killer won't kick in. That is caused by the time between releasing swap and storing swapped data inside the hard disk. If that is not gonna happen for a certain amount of times, the OOM killer kicks. By reducing the throttling interval, we simply pause more time over 256MB of memory, but an even slower machine won't fit into these intervals of 64KB.

There's not a "clean" solution, but we can use the v2 cgroup support for memory.reclaim and request enough data while we are swapping it.

On v1 we can either disable the cgroup OOM killer via memory.oom_control so the child blocks instead of being killed, or keep the loop unthrottled (the kernel already blocks faults in direct reclaim at memory.max) and report an OOM kill as TBROK via the child's exit status.

Regards,
--
Andrea Cervesato
SUSE QE Automation Engineer Linux
[email protected]

Signature · 1 line
Mailing list info: https://lists.linux.it/listinfo/ltp
#3 lore raw
Quoted 19 lines · Andrea Cervesato

Hi Wei,

Quoted 4 lines

Reduce the throttling interval from 2 MB to 64 KB (usleep for 200us every 64 KB) in the page-dirtying loop. This smooths out allocation spikes and gives asynchronous swap writeback time to catch up under slower VM storage environments, avoiding unexpected MemCG OOM-kills.

Unfortunately this is not a solution but a workaround, we need to make sure that we are able to see any change to the swap memory while allocating dirty pages close to the memory.max. This is probably a fix for a specific target rather than complete fix.

We need to find an another way to ensure that OOM killer won't kick in. That is caused by the time between releasing swap and storing swapped data inside the hard disk. If that is not gonna happen for a certain amount of times, the OOM killer kicks. By reducing the throttling interval, we simply pause more time over 256MB of memory, but an even slower machine won't fit into these intervals of 64KB.

Quoted 2 lines

There's not a "clean" solution, but we can use the v2 cgroup support for memory.reclaim and request enough data while we are swapping it.

The goal of this test is to reach memory.max and trigger kernel-driven swapping as a direct result of hitting that limit.
Proactively calling memory.reclaim forces swapping before reaching memory.max, meaning the memory limit is never actually reached. Furthermore, if calling memory.reclaim after hitting memory.max is too late to prevent OOM-kills on slow storage. So i do not think memory.reclaim solution can help on our oom failure.

Quoted 11 lines

On v1 we can either disable the cgroup OOM killer via memory.oom_control so the child blocks instead of being killed, or keep the loop unthrottled (the kernel already blocks faults in direct reclaim at memory.max) and report an OOM kill as TBROK via the child's exit status.

Regards,
--
Andrea Cervesato
SUSE QE Automation Engineer Linux
[email protected]

Signature · 1 line
Mailing list info: https://lists.linux.it/listinfo/ltp
#4 lore raw

Hi Wei,

Quoted 6 lines

The goal of this test is to reach memory.max and trigger kernel-driven swapping as a direct result of hitting that limit.
Proactively calling memory.reclaim forces swapping before reaching memory.max, meaning the memory limit is never actually reached. Furthermore, if calling memory.reclaim after hitting memory.max is too late to prevent OOM-kills on slow storage. So i do not think memory.reclaim solution can help on our oom failure.

memory.reclaim doesn't run during the pressure, but right after we reached the memory limit, so OOM killer can be handled by excluding it from the chain and getting an error instead of an OOM kill. In this way we know if there's an error while swapping, instead of guessing that device is too slow.

That's basically what we would do if we want to implement something that is suitable for all the devices: we disable OOM killer in the cgroup, we remove any sleep, we give a max runtime and if that is reached, it means that after a number of attempts we are still not swapping -> TFAIL.

Just notice that what I just described is not so far from what OOM-killer or memory.reclaim does. The ending result is exactly the same, but we do things by hand.

--
Andrea Cervesato
SUSE QE Automation Engineer Linux
[email protected]

Signature · 1 line
Mailing list info: https://lists.linux.it/listinfo/ltp
#5 lore raw
Quoted 15 lines · Andrea Cervesato

Hi Wei,

Quoted 6 lines

The goal of this test is to reach memory.max and trigger kernel-driven swapping as a direct result of hitting that limit.
Proactively calling memory.reclaim forces swapping before reaching memory.max, meaning the memory limit is never actually reached. Furthermore, if calling memory.reclaim after hitting memory.max is too late to prevent OOM-kills on slow storage. So i do not think memory.reclaim solution can help on our oom failure.

memory.reclaim doesn't run during the pressure, but right after we reached the memory limit, so OOM killer can be handled by excluding it from the chain and getting an error instead of an OOM kill. In this way we know if there's an error while swapping, instead of guessing that device is too slow.

Quoted 9 lines

That's basically what we would do if we want to implement something that is suitable for all the devices: we disable OOM killer in the cgroup, we remove any sleep, we give a max runtime and if that is reached, it means that after a number of attempts we are still not swapping -> TFAIL.

Just notice that what I just described is not so far from what OOM-killer or memory.reclaim does. The ending result is exactly the same, but we do things by hand.

Thanks for the technical discussion, and we should find better solution for avoid oom, but on cgroupv2(does not support disabling the OOM killer) scenario, replacing sleep-based pacing with memory.reclaim is simply replacing one guess with another. In user-space, it is impossible to catch the exact second when memory hits memory.max. Furthermore, calling memory.reclaim proactively just guesses when to bypass memory.max altogether, which invalidates the core goal of the test.

Quoted 4 lines

--
Andrea Cervesato
SUSE QE Automation Engineer Linux
[email protected]

Signature · 1 line
Mailing list info: https://lists.linux.it/listinfo/ltp
#6 lore raw
Quoted 7 lines

Thanks for the technical discussion, and we should find better solution for avoid oom, but on cgroupv2(does not support disabling the OOM killer) scenario, replacing sleep-based pacing with memory.reclaim is simply replacing one guess with another. In user-space, it is impossible to catch the exact second when memory hits memory.max. Furthermore, calling memory.reclaim proactively just guesses when to bypass memory.max altogether, which invalidates the core goal of the test.

Less LLM please :-)

The OOM killer can be disabled in v2, but only for a specific group.

If you don't want to use memory.reclaim in v2 / disable OOM killer in v1, then we need to do things by hand as explained in the previous comment, for both v1 and v2.

--
Andrea Cervesato
SUSE QE Automation Engineer Linux
[email protected]

Signature · 1 line
Mailing list info: https://lists.linux.it/listinfo/ltp
#7 lore raw
Quoted 9 lines · Andrea Cervesato
Quoted 7 lines

Thanks for the technical discussion, and we should find better solution for avoid oom, but on cgroupv2(does not support disabling the OOM killer) scenario, replacing sleep-based pacing with memory.reclaim is simply replacing one guess with another. In user-space, it is impossible to catch the exact second when memory hits memory.max. Furthermore, calling memory.reclaim proactively just guesses when to bypass memory.max altogether, which invalidates the core goal of the test.

Less LLM please :-)

I just use AI polish my sentences; the idea is mine :)

Quoted 2 lines

The OOM killer can be disabled in v2, but only for a specific group.

The following document shows: cgroupv2 has memory.oom.group, but is not used to disable the OOM killer.

Documentation/admin-guide/cgroup-v2.rst
memory.oom.group
A read-write single value file which exists on non-root cgroups. The default value is "0".

Determines whether the cgroup should be treated as an indivisible workload by the OOM killer. If set, all tasks belonging to the cgroup or to its descendants (if the memory cgroup is not a leaf cgroup) are killed together or not at all. This can be used to avoid partial kills to guarantee workload integrity. <<<<<<<

Quoted 4 lines

If you don't want to use memory.reclaim in v2 / disable OOM killer in v1, then we need to do things by hand as explained in the previous comment, for both v1 and v2.

I guess "need to do things by hand" includes totally removing the hard sleep time; currently, I have no good solution (as I mentioned in previous comments, memory.reclaim is not an option).

But I don't object to disabling the OOM solution, as cgroup-v2.rst also mentioned, we can protect task with oom_score_adj set to -1000, this is maybe another option for fix our openqa failed case. Do you think this is the preferred one?

Quoted 5 lines

--
Andrea Cervesato
SUSE QE Automation Engineer Linux
[email protected]

Signature · 1 line
Mailing list info: https://lists.linux.it/listinfo/ltp
#8 lore raw
Quoted 3 lines

I guess "need to do things by hand" includes totally removing the hard sleep time; currently, I have no good solution (as I mentioned in previous comments, memory.reclaim is not an option).

yes, basically we re-implement some sort of OOM-killer/memory.reclaim inside the code, instead of using them directly. That means we need to disable OOM killer, loop over the current swap status and eventually raise an error if after N iterations pages are not swapped.

But I feel it's not deterministic because on a really slow device we might run out of iterations. I still think memory.reclaim is the best choice for v2 in this case, because it will wait until a certain amount of memory has been reclaimed and eventually raise an error.

@Cyril do you have any idea on how to proceed here?

--
Andrea Cervesato
SUSE QE Automation Engineer Linux
[email protected]

Signature · 1 line
Mailing list info: https://lists.linux.it/listinfo/ltp
#9 lore raw

Hi!

Quoted 11 lines

yes, basically we re-implement some sort of OOM-killer/memory.reclaim inside the code, instead of using them directly. That means we need to disable OOM killer, loop over the current swap status and eventually raise an error if after N iterations pages are not swapped.

But I feel it's not deterministic because on a really slow device we might run out of iterations. I still think memory.reclaim is the best choice for v2 in this case, because it will wait until a certain amount of memory has been reclaimed and eventually raise an error.

@Cyril do you have any idea on how to proceed here?

If the goal is to make sure that kernel swaps when memory.max is reached we may as well slow down the dirtying loop so that it dirties memory with speed 20MB/s when it gets close to the limit.

Something as:

testcases/kernel/mem/mmapstress/mmapstress06.c +3 -2
index 4e6023399..2c0c6fb02 100644
@@ -59,8 +59,9 @@ static void run_test(void)
 
                for (size_t i = 0; i < map_size; i += page_size) {
                        mmapaddr[i] = 'a';
-                       if ((i % (2 * 1024 * 1024)) == 0)
-                               usleep(1000);
+
+                       if (i >= mem_limit && !(i % (2 * 1024 * 1024)))
+                               usleep(100000);
                }
 

The testrun increses to 6s from 1s but it makes the OOM very unlikely to happen.

Signature · 5 lines
Cyril Hrubis
[email protected]

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp
#10 lore raw
Quoted 34 lines · Cyril Hrubis

Hi!

Quoted 11 lines

yes, basically we re-implement some sort of OOM-killer/memory.reclaim inside the code, instead of using them directly. That means we need to disable OOM killer, loop over the current swap status and eventually raise an error if after N iterations pages are not swapped.

But I feel it's not deterministic because on a really slow device we might run out of iterations. I still think memory.reclaim is the best choice for v2 in this case, because it will wait until a certain amount of memory has been reclaimed and eventually raise an error.

@Cyril do you have any idea on how to proceed here?

If the goal is to make sure that kernel swaps when memory.max is reached we may as well slow down the dirtying loop so that it dirties memory with speed 20MB/s when it gets close to the limit.

Something as:

testcases/kernel/mem/mmapstress/mmapstress06.c +3 -2
index 4e6023399..2c0c6fb02 100644
@@ -59,8 +59,9 @@ static void run_test(void)
 
                for (size_t i = 0; i < map_size; i += page_size) {
                        mmapaddr[i] = 'a';
-                       if ((i % (2 * 1024 * 1024)) == 0)
-                               usleep(1000);
+
+                       if (i >= mem_limit && !(i % (2 * 1024 * 1024)))
+                               usleep(100000);
                }
 
Quoted 2 lines

The testrun increses to 6s from 1s but it makes the OOM very unlikely to happen.

Thanks for your suggestion.
I have tested above solution in our openqa setup but still encounter failure, 3 test cases still failure with oom after run 100 cases. Maybe slow down dirty loop
is still too late?

Quoted 4 lines
Signature · 3 lines
-- 
Cyril Hrubis
[email protected]
Signature · 1 line
Mailing list info: https://lists.linux.it/listinfo/ltp
#11 lore raw

Hi Wei,

Quoted 17 lines
Quoted 17 lines

Something as:

testcases/kernel/mem/mmapstress/mmapstress06.c +3 -2
index 4e6023399..2c0c6fb02 100644
@@ -59,8 +59,9 @@ static void run_test(void)
 
                for (size_t i = 0; i < map_size; i += page_size) {
                        mmapaddr[i] = 'a';
-                       if ((i % (2 * 1024 * 1024)) == 0)
-                               usleep(1000);
+
+                       if (i >= mem_limit && !(i % (2 * 1024 * 1024)))
+                               usleep(100000);
                }
 
Quoted 6 lines
Quoted 2 lines

The testrun increses to 6s from 1s but it makes the OOM very unlikely to happen.

Thanks for your suggestion.
I have tested above solution in our openqa setup but still encounter failure, 3 test cases still failure with oom after run 100 cases. Maybe slow down dirty loop
is still too late?

I still believe there's no way to avoid this, unless we re-implement the OOM-killer or memory.reclaim mechanism increasing the amount of attempts per request. We will always encounter hardware that is too slow that won't swap memory fast enough for the test.

--
Andrea Cervesato
SUSE QE Automation Engineer Linux
[email protected]

Signature · 1 line
Mailing list info: https://lists.linux.it/listinfo/ltp
#12 lore raw
Quoted 26 lines · Andrea Cervesato

Hi Wei,

Quoted 17 lines
Quoted 17 lines

Something as:

testcases/kernel/mem/mmapstress/mmapstress06.c +3 -2
index 4e6023399..2c0c6fb02 100644
@@ -59,8 +59,9 @@ static void run_test(void)
 
                for (size_t i = 0; i < map_size; i += page_size) {
                        mmapaddr[i] = 'a';
-                       if ((i % (2 * 1024 * 1024)) == 0)
-                               usleep(1000);
+
+                       if (i >= mem_limit && !(i % (2 * 1024 * 1024)))
+                               usleep(100000);
                }
 
Quoted 6 lines
Quoted 2 lines

The testrun increses to 6s from 1s but it makes the OOM very unlikely to happen.

Thanks for your suggestion.
I have tested above solution in our openqa setup but still encounter failure, 3 test cases still failure with oom after run 100 cases. Maybe slow down dirty loop
is still too late?

Quoted 4 lines

I still believe there's no way to avoid this, unless we re-implement the OOM-killer or memory.reclaim mechanism increasing the amount of attempts per request. We will always encounter hardware that is too slow that won't swap memory fast enough for the test.

@Andrea @Cyril
As my former email suggestion "protect task with oom_score_adj set to -1000", and this can work after 100 runs. If you agree this solution then i will create v2 new patch.

testcases/kernel/mem/mmapstress/mmapstress06.c +8 -4
@@ -23,6 +23,7 @@
 #include <errno.h>
 #include <stdlib.h>
 #include "tst_test.h"
+#include "tst_memutils.h"
 
 static struct tst_cg_group *cg_child;
 static int page_size;
@@ -47,6 +48,8 @@ static void run_test(void)
                char *mmapaddr;
                unsigned long cg_swap_before = 0, cg_swap_after = 0;
 
+               tst_enable_oom_protection(0);
+
                /* Move child to the constrained cgroup */
                SAFE_CG_PRINTF(cg_child, "cgroup.procs", "%d", getpid());
 
@@ -57,11 +60,12 @@ static void run_test(void)
 
                tst_res(TINFO, "Dirtying %zu bytes in child", map_size);
 
-               for (size_t i = 0; i < map_size; i += page_size) {
+               /*
+                * Tasks with the OOM protection (oom_score_adj set to -1000)
+                * are treated as an exception and are never killed.
+                */
+               for (size_t i = 0; i < map_size; i += page_size)
                        mmapaddr[i] = 'a';
-                       if ((i % (2 * 1024 * 1024)) == 0)
-                               usleep(1000);
-               }
 
                SAFE_CG_SCANF(cg_child, "memory.swap.current", "%lu", &cg_swap_after);
 
Quoted 5 lines

--
Andrea Cervesato
SUSE QE Automation Engineer Linux
[email protected]

Signature · 1 line
Mailing list info: https://lists.linux.it/listinfo/ltp
#13 lore raw
In reply to Wei Gao via ltp

Hi!

Quoted 15 lines
Quoted 15 lines
testcases/kernel/mem/mmapstress/mmapstress06.c +3 -2
index 4e6023399..2c0c6fb02 100644
@@ -59,8 +59,9 @@ static void run_test(void)
 
                for (size_t i = 0; i < map_size; i += page_size) {
                        mmapaddr[i] = 'a';
-                       if ((i % (2 * 1024 * 1024)) == 0)
-                               usleep(1000);
+
+                       if (i >= mem_limit && !(i % (2 * 1024 * 1024)))
+                               usleep(100000);
                }
 
Quoted 6 lines
Quoted 2 lines

The testrun increses to 6s from 1s but it makes the OOM very unlikely to happen.

Thanks for your suggestion.
I have tested above solution in our openqa setup but still encounter failure, 3 test cases still failure with oom after run 100 cases. Maybe slow down dirty loop
is still too late?

AFAIK the swapping does not start until we are over the limit, so slowing it down before we reach the limit is not going to change anything. For v2 cgroup we can also set a soft limit memory.high to slightly smaller number than memory.max, whith that the kernel would start the swapping once memory.high was reached.

Something as:

if (!TST_CG_VER_IS_V1(cg_child, memory))

        SAFE_CG_PRINT(cg_child, "memory.high", "%lu", mem_limit - mem_limit/8);
Signature · 5 lines
Cyril Hrubis
[email protected]

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp
#14 lore raw

PATCH v2 1/1 mmapstress06: Use memory.high in cgroup v2 to start swapping early

In reply to Wei Gao via ltp

Set memory.high to slightly smaller than memory.max in cgroup v2 to start swapping earlier. In cgroup v2, the asynchronous swap writeback does not start until we are over the limit (memory.max). Setting memory.high starts the reclaiming/swapping once memory.high is reached, preventing the child process from hitting the memory.max limit and getting killed by the OOM-killer.

---
v1->v2:
- Update solution base Cyril's suggestion

 lib/tst_cgroup.c                               | 1 +
 testcases/kernel/mem/mmapstress/mmapstress06.c | 7 +++++--
 2 files changed, 6 insertions(+), 2 deletions(-)
lib/tst_cgroup.c +1 -0
index 7c6ecd4d2..3ee9d32a4 100644
@@ -180,6 +180,7 @@ static const struct cgroup_file memory_ctrl_files[] = {
 	{ "memory.events", NULL, CTRL_MEMORY },
 	{ "memory.low", NULL, CTRL_MEMORY },
 	{ "memory.min", NULL, CTRL_MEMORY },
+	{ "memory.high", "memory.soft_limit_in_bytes", CTRL_MEMORY },
 	{ "memory.max", "memory.limit_in_bytes", CTRL_MEMORY },
 	{ "memory.stat", "memory.stat", CTRL_MEMORY },
 	{ "memory.swappiness", "memory.swappiness", CTRL_MEMORY },
testcases/kernel/mem/mmapstress/mmapstress06.c +5 -2
index 4e6023399..7cc17fc75 100644
@@ -37,10 +37,13 @@ static void run_test(void)
 	/* Set memory limit to force swapping of the mapping */
 	SAFE_CG_PRINTF(cg_child, "memory.max", "%lu", (unsigned long)mem_limit);
 
-	if (TST_CG_VER_IS_V1(cg_child, "memory"))
+	if (TST_CG_VER_IS_V1(cg_child, "memory")) {
 		SAFE_CG_PRINT(cg_child, "memory.swap.max", "-1");
-	else
+	} else {
 		SAFE_CG_PRINT(cg_child, "memory.swap.max", "max");
+		SAFE_CG_PRINTF(cg_child, "memory.high", "%lu",
+			       (unsigned long)(mem_limit - mem_limit / 8));
+	}
 
 	child_pid = SAFE_FORK();
 	if (!child_pid) {
Signature · 5 lines
2.55.0


-- 
Mailing list info: https://lists.linux.it/listinfo/ltp
#15 lore raw

Re: [LTP] mmapstress06: Use memory.high in cgroup v2 to start swapping early

Hi Wei,

Quoted 1 line · Wei Gao

mmapstress06: Use memory.high in cgroup v2 to start swapping early

Verdict - Reviewed

---
Note:

The agent can sometimes produce false positives although often its findings are genuine. If you find issues with the review, please comment this email or ignore the suggestions.

Regards,
LTP AI Reviewer

Signature · 1 line
Mailing list info: https://lists.linux.it/listinfo/ltp