OOM protection for shell tests
v2 cover OOM protection for shell tests
Under Li's idea, implement a OOM protection mechanism for the shell tests so we can avoid OOM for memcg stress tests.
---
Changes in v2:
- update shell OOM protection functional test
- Link to v1: https://lore.kernel.org/[email protected]
---
Andrea Cervesato (2):
shell: add optional OOM protection
memcg_stress: survive OOM by targeting the stressors
doc/developers/writing_tests.rst | 3 ++ .../controllers/memcg/stress/memcg_stress_test.sh | 1 + testcases/lib/tests/shell_oom_protection.sh | 42 ++++++++++++++++ testcases/lib/tst_test.sh | 57 +++++++++++++++++++++- 4 files changed, 102 insertions(+), 1 deletion(-)
---
base-commit: 4918812af24a0e06fa3a9ba6da2679cfea1f2126
change-id: 20260713-shell_oom_protection-6221ab825220
Best regards,
Signature · 5 lines
Andrea Cervesato <[email protected]> -- Mailing list info: https://lists.linux.it/listinfo/ltp
PATCH v2 1/2 shell: add optional OOM protection
From: Andrea Cervesato <[email protected]>
Add TST_OOM_PROTECTION to activate OOM protection in shell tests. When enabled, the shell harness shields itself from the OOM killer and runs the test in a child process, so it survives memory pressure and can still report results (e.g. during memcg stress tests).
--- doc/developers/writing_tests.rst | 3 ++ testcases/lib/tests/shell_oom_protection.sh | 42 +++++++++++++++++++++ testcases/lib/tst_test.sh | 57 ++++++++++++++++++++++++++++- 3 files changed, 101 insertions(+), 1 deletion(-)
doc/developers/writing_tests.rst +3 -0
@@ -549,6 +549,9 @@ LTP C And Shell Test API Comparison * - not applicable - TST_FS_TYPE + * - not applicable + - TST_OOM_PROTECTION + .. list-table:: :header-rows: 1
testcases/lib/tests/shell_oom_protection.sh new file +42 -0
42 lines not on this page · raw message
testcases/lib/tst_test.sh +56 -1
74 lines not on this page · raw message
Signature · 5 lines
2.51.0 -- Mailing list info: https://lists.linux.it/listinfo/ltp
Re: [LTP] shell: add optional OOM protection
Hi Andrea,
Quoted 1 line · Andrea Cervesato
shell: add optional OOM protection
--- [PATCH 1/2] ---
testcases/lib/tests/shell_oom_protection.sh
Quoted 2 lines
testcases/lib/tests/shell_oom_protection.sh new file +0
-0
The lib self-tests are launched from testcases/lib/run_tests.sh through the explicit TESTS_PASS / TESTS_FAIL / TESTS_TBROK / TESTS_TCONF lists. This new test is not added to any of them, so how is it expected to run in the self-test suite?
As a root run it emits two TPASS (exit 0), which would place it in TESTS_PASS.
Quoted 12 lines
testcases/lib/tests/shell_oom_protection.sh new file +12
-0+# doc +# Verify that TST_OOM_PROTECTION shields the shell harness from the OOM +# killer and resets the test process oom_score_adj to 0. +# +# --- +# env +# { +# } +# --- + +TST_TESTFUNC=do_test +TST_OOM_PROTECTION=1
[...]
Quoted 2 lines
+.
tst_test.sh
+tst_run
The other tests in this directory use the metadata loader style (. tst_loader.sh ... . tst_run.sh), where the doc/env blocks are parsed. Here the test sources tst_test.sh, so these comment blocks are not parsed and become dead comments. The doc block is also missing its opening "# ---" delimiter, so it would be malformed under the metadata loader too.
Would it be cleaner to either convert to the tst_loader.sh/tst_run.sh style with a proper metadata block, or drop the metadata comments?
Quoted 4 lines
testcases/lib/tests/shell_oom_protection.sh new file +4
-0+ _tst_enable_oom_protection + + ( + _tst_disable_oom_protection
testcases/lib/tests/shell_oom_protection.sh
Quoted 10 lines
testcases/lib/tests/shell_oom_protection.sh new file +10
-0+ _TST_OOM_PROTECTION=0 + export _TST_OOM_PROTECTION + tst_run "$@" + ) & + _tst_pid=$! + + wait "$_tst_pid" + _tst_ret=$? + + if [ "$_tst_ret" -eq 137 ]; then
On the SIGKILL path the test runs in the child subshell, so its setup state (TST_DO_CLEANUP, TST_TMPDIR, TST_MOUNT_FLAG, TST_DEVICE_FLAG) lives in the killed child. The surviving harness reports TBROK but its tst_brk -> _tst_do_exit then cleans nothing, so external resources created by the child (tmpdir, mounts, devices, cgroups) are left behind.
This is largely pre-existing (an OOM-killed driver skipped cleanup before too), but now the process survives to report while the cleanup is lost. Is that acceptable, or should the surviving harness own the teardown? See the concrete impact under patch 2/2.
--- [PATCH 2/2] ---
Quoted 1 line
+TST_OOM_PROTECTION=1
The mechanism protects the harness (oom_score_adj=-1000) but runs the driver logic in a child with oom_score_adj=0, the same value as the memcg_process_stress stressors. In practice the stressors have the larger RSS and are the natural OOM victims, so the driver child survives.
If the driver child is nonetheless the OOM victim (the case the commit message describes), the harness reports TBROK "Test killed!" and cleanup() / cgroup_cleanup() never run, leaving the memcg mount and cgroup directories behind. Is that residual leak acceptable given the survival goal?
The Fixes: 02961a7b2bb4 tag is valid and matches local history.
Verdict - Needs revision
---
Note:
The agent can sometimes produce false positives although often its findings are genuine. If you find issues with the review, please comment this email or ignore the suggestions.
Regards,
LTP AI Reviewer
Signature · 1 line
Mailing list info: https://lists.linux.it/listinfo/ltp
Re: [LTP] [PATCH v2 1/2] shell: add optional OOM protection
In reply to Andrea CervesatoHi Andrea,
Quoted 41 lines
Add
TST_OOM_PROTECTIONto activate OOM protection in shell tests. When enabled, the shell harness shields itself from the OOM killer and runs the test in a child process, so it survives memory pressure and can still report results (e.g. during memcg stress tests).testcases/lib/tests/shell_oom_protection.sh new file +30
-0@@ -0,0 +1,42 @@ +#!/bin/sh +# SPDX-License-Identifier: GPL-2.0-or-later +# Copyright (c) 2026 Linux Test Project +# +# doc +# Verify that TST_OOM_PROTECTION shields the shell harness from the OOM +# killer and resets the test process oom_score_adj to 0. +# +# --- +# env +# { +# } +# --- + +TST_TESTFUNC=do_test +TST_OOM_PROTECTION=1 + +read_oom_score_adj() { + cat "/proc/$1/oom_score_adj" 2>/dev/null +} + +do_test() { + local self_score harness_score + + self_score=$(read_oom_score_adj self) + harness_score=$(read_oom_score_adj "$$") + + if [ "$self_score" = 0 ]; then + tst_res TPASS "test process has oom_score_adj reset to 0" + else
testcases/lib/tests/shell_oom_protection.sh line 40
Quoted 10 lines
testcases/lib/tests/shell_oom_protection.sh new file +10
-0+ tst_res TFAIL "test process oom_score_adj is $self_score, expected 0" + fi + + if [ "$harness_score" = -1000 ]; then + tst_res TPASS "shell harness is protected from OOM" + else + tst_res TCONF "shell harness OOM protection unavailable" + fi +} +
If the test always TPASS or TCONF, please add it to lib/newlib_tests/runtest.sh (to be run in CI).
...
Quoted 8 lines
testcases/lib/tests/shell_oom_protection.sh new file +2
-0+. tst_test.sh +tst_runtestcases/lib/tst_test.sh +0
-0@@ -28,6 +28,57 @@ export TST_USR_GID="${LTP_USR_GID:-65534}" trap "tst_brk TBROK 'test interrupted'" INT
testcases/lib/tst_test.sh export TST_USR_GID="${LTP_USR_GID:-65534}" · line 38
Quoted 10 lines
testcases/lib/tst_test.sh +8
-0trap "unset _tst_setup_timer_pid; tst_brk TBROK 'test terminated'" TERM +_tst_set_oom_score_adj() +{ + local value="$1" + local path="/proc/self/oom_score_adj" + + [ -e "$path" ] || return 0 + + echo "$value" > "$path" 2>/dev/null || return 0
So, we want to hide "write error: Permission denied"?
Quoted 14 lines
testcases/lib/tst_test.sh +14
-0+} + +_tst_enable_oom_protection() +{ + _tst_set_oom_score_adj -1000 +} + +_tst_disable_oom_protection() +{ + _tst_set_oom_score_adj 0 +} + +_tst_run_oom_protected() +{
testcases/lib/tst_test.sh export TST_USR_GID="${LTP_USR_GID:-65534}" · line 62
Quoted 10 lines
testcases/lib/tst_test.sh +10
-0+ local _tst_pid + local _tst_ret + + # Shield the harness from the OOM killer and run the test in a child. + # The child keeps the default oom_score_adj so that it, and any process + # it spawns, stay killable under memory pressure while the harness + # survives to report results. + _tst_enable_oom_protection + + (
Interesting idea. Hopefully it could work (I'm not sure if tst_test.sh API will work well with yet another subshell, if killing will work correctly).
testcases/lib/tst_test.sh export TST_USR_GID="${LTP_USR_GID:-65534}" · line 65
Quoted 3 lines
testcases/lib/tst_test.sh +3
-0+ _tst_disable_oom_protection + _TST_OOM_PROTECTION=0 + export _TST_OOM_PROTECTION
NOTE: you can export variable with value in a single command:
export _TST_OOM_PROTECTION=0
Also, "$_TST_" does not have the protection about misuse which "$TST_" has (that protection for _tst_i in $(grep ... you modified).
Please use a different name with "$TST_". But best would be just to assign a different value:
export TST_OOM_PROTECTION=2
testcases/lib/tst_test.sh export TST_USR_GID="${LTP_USR_GID:-65534}" · line 66
Quoted 1 line
testcases/lib/tst_test.sh +1
-0+ tst_run "$@"
Running the test in the cleanup phase, that looks to me strange.
Quoted 19 lines
testcases/lib/tst_test.sh +15
-0+ ) & + _tst_pid=$! + + wait "$_tst_pid" + _tst_ret=$? + + if [ "$_tst_ret" -eq 137 ]; then + tst_res TINFO "Test was SIGKILLed: OOM killer or timeout?" + tst_res TINFO "On a slow machine try exporting LTP_TIMEOUT_MUL > 1" + tst_brk TBROK "Test killed!" + fi + + exit "$_tst_ret" +} + _tst_do_cleanup() { if [ -n "$TST_DO_CLEANUP" -a -n "$TST_CLEANUP" -a -z "$LTP_NO_CLEANUP" ]; then
testcases/lib/tst_test.sh tst_run() · line 734
Quoted 4 lines
testcases/lib/tst_test.sh +1
-0@@ -680,12 +731,16 @@ tst_run() local _tst_pattern='[='\''"} \t\/:`$\;|].*' local ret + if [ "$TST_OOM_PROTECTION" = 1 -a "$_TST_OOM_PROTECTION" != 0 ]; then
And here have just:
if [ "$TST_OOM_PROTECTION" = 1 ]; then
Kind regards,
Petr
Quoted 13 lines
testcases/lib/tst_test.sh +4
-1+ _tst_run_oom_protected "$@" + fi + if [ -n "$TST_TEST_PATH" ]; then for _tst_i in $(grep '^[^#]*\<TST_' "$TST_TEST_PATH" | sed "s/.*TST_//; s/$_tst_pattern//"); do case "$_tst_i" in ALL_FILESYSTEMS|DISABLE_APPARMOR|DISABLE_SELINUX);; SETUP|CLEANUP|TESTFUNC|ID|CNT|MIN_KVER);; - OPTS|USAGE|PARSE_ARGS|POS_ARGS);; + OPTS|USAGE|PARSE_ARGS|POS_ARGS|OOM_PROTECTION);; NEEDS_ROOT|NEEDS_TMPDIR|TMPDIR|NEEDS_DEVICE|DEVICE);; NEEDS_CMDS|NEEDS_MODULE|MODPATH|DATAROOT);; NEEDS_DRIVERS|FS_TYPE|MNTPOINT|MNT_PARAMS);;
Signature · 1 line
Mailing list info: https://lists.linux.it/listinfo/ltp
Re: [LTP] [PATCH v2 1/2] shell: add optional OOM protection
In reply to Andrea CervesatoQuoted 61 lines · Andrea Cervesato
testcases/lib/tst_test.sh +51
-0@@ -28,6 +28,57 @@ export TST_USR_GID="${LTP_USR_GID:-65534}" trap "tst_brk TBROK 'test interrupted'" INT trap "unset _tst_setup_timer_pid; tst_brk TBROK 'test terminated'" TERM +_tst_set_oom_score_adj() +{ + local value="$1" + local path="/proc/self/oom_score_adj" + + [ -e "$path" ] || return 0 + + echo "$value" > "$path" 2>/dev/null || return 0 +} + +_tst_enable_oom_protection() +{ + _tst_set_oom_score_adj -1000 +} + +_tst_disable_oom_protection() +{ + _tst_set_oom_score_adj 0 +} + +_tst_run_oom_protected() +{ + local _tst_pid + local _tst_ret + + # Shield the harness from the OOM killer and run the test in a child. + # The child keeps the default oom_score_adj so that it, and any process + # it spawns, stay killable under memory pressure while the harness + # survives to report results. + _tst_enable_oom_protection + + ( + _tst_disable_oom_protection + _TST_OOM_PROTECTION=0 + export _TST_OOM_PROTECTION + tst_run "$@" + ) & + _tst_pid=$! + + wait "$_tst_pid" + _tst_ret=$? + + if [ "$_tst_ret" -eq 137 ]; then + tst_res TINFO "Test was SIGKILLed: OOM killer or timeout?" + tst_res TINFO "On a slow machine try exporting LTP_TIMEOUT_MUL > 1" + tst_brk TBROK "Test killed!" + fi + + exit "$_tst_ret" +} + _tst_do_cleanup() { if [ -n "$TST_DO_CLEANUP" -a -n "$TST_CLEANUP" -a -z "$LTP_NO_CLEANUP" ]; then
testcases/lib/tst_test.sh tst_run() · line 736
Quoted 6 lines
testcases/lib/tst_test.sh +3
-0@@ -680,12 +731,16 @@ tst_run() local _tst_pattern='[='\''"} \t\/:`$\;|].*' local ret + if [ "$TST_OOM_PROTECTION" = 1 -a "$_TST_OOM_PROTECTION" != 0 ]; then + _tst_run_oom_protected "$@" + fi
In the C library, OOM protection for the LTP library and harness is enabled by default. But in such Shell implementation, we have to explicitly enable it via 'TST_OOM_PROTECTION=1'.
If we keep them consistent and remove the TST_OOM_PROTECTION variable, things will become easier I guess :).
Signature · 5 lines
Regards, Li Wang -- Mailing list info: https://lists.linux.it/listinfo/ltp
Re: [LTP] [PATCH v2 1/2] shell: add optional OOM protection
Quoted 7 lines
testcases/lib/tst_test.sh tst_run() · line 736
Quoted 3 lines
testcases/lib/tst_test.sh +3
-0+ if [ "$TST_OOM_PROTECTION" = 1 -a "$_TST_OOM_PROTECTION" != 0 ]; then + _tst_run_oom_protected "$@" + fiIn the C library, OOM protection for the LTP library and harness is enabled by default. But in such Shell implementation, we have to explicitly enable it via 'TST_OOM_PROTECTION=1'.
Say precisely: only protect LTP library not contains upper harness. The upper harness can to be handled out from LTP.
See:
commit 8e5fe1d5f2 ("lib: enable OOM protection for ltp lib process")
Quoted 2 lines
If we keep them consistent and remove the
TST_OOM_PROTECTIONvariable, things will become easier I guess :).
Signature · 5 lines
Regards, Li Wang -- Mailing list info: https://lists.linux.it/listinfo/ltp
PATCH v2 2/2 memcg_stress: survive OOM by targeting the stressors
In reply to Andrea CervesatoFrom: Andrea Cervesato <[email protected]>
The single-cgroup subtest intentionally lets one process fault in the whole memory budget to push the system to its memory ceiling, so hitting the OOM killer is an expected part of the stress. The problem is that the OOM killer could reap the driver script, turning that expected pressure into a spurious TBROK.
Activate OOM protection so the driver script won't be the OOM victim.
--- testcases/kernel/controllers/memcg/stress/memcg_stress_test.sh | 1 + 1 file changed, 1 insertion(+)
testcases/kernel/controllers/memcg/stress/memcg_stress_test.sh +1 -0
@@ -14,6 +14,7 @@ TST_CLEANUP=cleanup TST_CNT=2 TST_NEEDS_ROOT=1 TST_NEEDS_CMDS="mount umount cat kill mkdir rmdir grep awk cut" +TST_OOM_PROTECTION=1 # Each test case runs for 900 secs when everything fine # therefore the default 5 mins timeout is not enough.
Signature · 5 lines
2.51.0 -- Mailing list info: https://lists.linux.it/listinfo/ltp
Hi Li,
can you please take a look at this patch?
Thanks,
--
Andrea Cervesato
SUSE QE Automation Engineer Linux
[email protected]
Signature · 1 line
Mailing list info: https://lists.linux.it/listinfo/ltp