Skip to contentSkip to conversation

shell: enable OOM protection by default

PATCH In review Andrea Cervesato 5 participants 7 messages Aug 10 – Aug 11

lore
Share & tools
lore
#0 lore raw

PATCH v4 1/1 shell: enable OOM protection by default

From: Andrea Cervesato <[email protected]>

The shell harness shields itself from the OOM killer and runs the test in a child process, so it survives memory pressure and can still report results (e.g. during memcg stress tests).

---
Under Li's idea, implement a OOM protection mechanism for the shell
tests so we can avoid OOM for memcg stress tests.
---
Changes in v4:
- fix with Petr suggestions
- fix with AI agent suggestions
- Link to v3: https://lore.kernel.org/[email protected]

Changes in v3:
- simplify oom code
- enable OOM protection by default
- Link to v2: https://lore.kernel.org/[email protected]

Changes in v2:
- update shell OOM protection functional test
- Link to v1: https://lore.kernel.org/[email protected]

To: Linux Test Project <[email protected]>

---
 lib/newlib_tests/runtest.sh                  |  1 +
 lib/newlib_tests/shell/tst_oom_protection.sh | 34 +++++++++++
 testcases/lib/tst_test.sh                    | 84 ++++++++++++++++++++++++++--
 3 files changed, 115 insertions(+), 4 deletions(-)
lib/newlib_tests/runtest.sh +1 -0
index 71808ef8b..7e2d0a2ac 100755
@@ -44,6 +44,7 @@ shell/tst_check_driver.sh
 shell/tst_check_kconfig0[1-5].sh
 shell/tst_mount_device.sh
 shell/tst_mount_device_tmpfs.sh
+shell/tst_oom_protection.sh
 shell/tst_skip_filesystems.sh
 shell/net/*.sh}"
 
lib/newlib_tests/shell/tst_oom_protection.sh new file +34 -0
new file mode 100755
index 000000000..564680700
@@ -0,0 +1,34 @@
+#!/bin/sh
+# SPDX-License-Identifier: GPL-2.0-or-later
+# Copyright (c) 2026 Linux Test Project
+
+TST_TESTFUNC=do_test
+
+read_oom_score_adj() {
+	cat "/proc/$1/oom_score_adj"
+}
+
+do_test() {
+	local harness_score body_score
+
+	# $$ points to the protected harness, while /proc/self is the
+	# unprotected child that actually runs the test body.
+	harness_score=$(read_oom_score_adj "$$")
+	body_score=$(read_oom_score_adj self)
+
+	if [ "$harness_score" != -1000 ]; then
+		tst_res TCONF "shell harness OOM protection unavailable"
+		return
+	fi
+
+	tst_res TPASS "shell harness is protected from OOM by default"
+
+	if [ "$body_score" = 0 ]; then
+		tst_res TPASS "test body runs in an unprotected child (oom_score_adj=0)"
+	else
+		tst_res TFAIL "test body oom_score_adj is $body_score, expected 0"
+	fi
+}
+
+. tst_test.sh
+tst_run
testcases/lib/tst_test.sh +80 -4

108 lines not on this page · raw message

---
base-commit: ee945bb71ba7015dace7d983cf98a530b3eb53a4
change-id: 20260713-shell_oom_protection-6221ab825220

Best regards,

Signature · 5 lines
Andrea Cervesato <[email protected]>


-- 
Mailing list info: https://lists.linux.it/listinfo/ltp
#1 lore raw

Hi Andrea,

testcases/lib/tst_test.sh export TST_USR_GID="${LTP_USR_GID:-65534}" · line 43

Quoted 5 lines · Andrea Cervesato

shell: enable OOM protection by default

testcases/lib/tst_test.sh +3 -0
+	if ! echo "$value" > "$path"; then
+		tst_res TWARN "Can't adjust score"
+	fi

Could this avoid reporting TWARN when CAP_SYS_RESOURCE is absent, as tst_memutils.c does? Lowering oom_score_adj requires that capability, so every unprivileged shell test now gets a warning result. The new self-test itself reports TCONF but exits 36 because this TWARN remains counted.

Quoted 4 lines
testcases/lib/tst_test.sh +4 -0
+	(
+		TST_CHILD=1
+		_tst_disable_oom_protection
+		trap '_tst_cleanup_timer; _tst_write_results' EXIT

testcases/lib/tst_test.sh tst_run() · line 863

Quoted 10 lines
testcases/lib/tst_test.sh +10 -0
+
+		if [ "$TST_ALL_FILESYSTEMS" = 1 ]; then
+			_tst_run_tcases_per_fs
+		else
+			_tst_run_iterations
+		fi
+	) &
+
+	wait $!
+	ret=$?

How can the protected parent perform cleanup if this child is OOM-killed or times out? TST_DO_CLEANUP and TST_MOUNT_FLAG are set only in the subshell. On an abnormal child exit, the parent therefore reports TBROK without calling TST_CLEANUP or unmounting a device mounted by _prepare_device(), leaving external test state behind.

Verdict - Needs revision

---
Note:

The agent can sometimes produce false positives although often its findings are genuine. If you find issues with the review, please comment this email or ignore the suggestions.

Regards,
LTP AI Reviewer

Signature · 1 line
Mailing list info: https://lists.linux.it/listinfo/ltp
#2 lore raw

Hi Andrea,

Quoted 5 lines

Hi Andrea,

Quoted 1 line · Andrea Cervesato

shell: enable OOM protection by default

Quoted 8 lines

testcases/lib/tst_test.sh export TST_USR_GID="${LTP_USR_GID:-65534}" · line 43

Quoted 3 lines
testcases/lib/tst_test.sh +3 -0
+	if ! echo "$value" > "$path"; then
+		tst_res TWARN "Can't adjust score"
+	fi

Could this avoid reporting TWARN when CAP_SYS_RESOURCE is absent, as tst_memutils.c does? Lowering oom_score_adj requires that capability, so every unprivileged shell test now gets a warning result. The new self-test itself reports TCONF but exits 36 because this TWARN remains counted.

Ah, this really needs to be guarded behind if [ "$(id -ru)" != 0 ]; then

If you're in a mood for it, you could factor out [ "$(id -ru)" != 0 ] check into it's own function tst_check_root or tst_is_root which would be then used in tst_require_root and in this function.

I repeat my suggestion from last revision: add a helper into testcases/lib which will just call C function. That way the behavior will be the same (use capabilities). That would be more clearer solution.

Kind regards,
Petr

Quoted 12 lines
Quoted 12 lines
testcases/lib/tst_test.sh +12 -0
+	(
+		TST_CHILD=1
+		_tst_disable_oom_protection
+		trap '_tst_cleanup_timer; _tst_write_results' EXIT
+
+		if [ "$TST_ALL_FILESYSTEMS" = 1 ]; then
+			_tst_run_tcases_per_fs
+		else
+			_tst_run_iterations
+		fi
+	) &
+
Quoted 8 lines

testcases/lib/tst_test.sh tst_run() · line 863

Quoted 2 lines
testcases/lib/tst_test.sh +2 -0
+	wait $!
+	ret=$?

How can the protected parent perform cleanup if this child is OOM-killed or times out? TST_DO_CLEANUP and TST_MOUNT_FLAG are set only in the subshell. On an abnormal child exit, the parent therefore reports TBROK without calling TST_CLEANUP or unmounting a device mounted by _prepare_device(), leaving external test state behind.

Sounds serious :). I'll comment on the patch itself.

Kind regards,
Petr

Quoted 11 lines

Verdict - Needs revision

---
Note:

The agent can sometimes produce false positives although often its findings are genuine. If you find issues with the review, please comment this email or ignore the suggestions.

Regards,
LTP AI Reviewer

Signature · 1 line
Mailing list info: https://lists.linux.it/listinfo/ltp
#3 lore raw
In reply to Andrea Cervesato

Hi Andrea,

...

Quoted 66 lines
lib/newlib_tests/shell/tst_oom_protection.sh new file +34 -0
new file mode 100755
index 000000000..564680700
@@ -0,0 +1,34 @@
+#!/bin/sh
+# SPDX-License-Identifier: GPL-2.0-or-later
+# Copyright (c) 2026 Linux Test Project
+
+TST_TESTFUNC=do_test
+
+read_oom_score_adj() {
+	cat "/proc/$1/oom_score_adj"
+}
+
+do_test() {
+	local harness_score body_score
+
+	# $$ points to the protected harness, while /proc/self is the
+	# unprotected child that actually runs the test body.
+	harness_score=$(read_oom_score_adj "$$")
+	body_score=$(read_oom_score_adj self)
+
+	if [ "$harness_score" != -1000 ]; then
+		tst_res TCONF "shell harness OOM protection unavailable"
+		return
+	fi
+
+	tst_res TPASS "shell harness is protected from OOM by default"
+
+	if [ "$body_score" = 0 ]; then
+		tst_res TPASS "test body runs in an unprotected child (oom_score_adj=0)"
+	else
+		tst_res TFAIL "test body oom_score_adj is $body_score, expected 0"
+	fi
+}
+
+. tst_test.sh
+tst_run
testcases/lib/tst_test.sh +18 -0
index b3e7e29bb..e42e562f5 100644
@@ -28,6 +28,48 @@ export TST_USR_GID="${LTP_USR_GID:-65534}"
 trap "tst_brk TBROK 'test interrupted'" INT
 trap "unset _tst_setup_timer_pid; tst_brk TBROK 'test terminated'" TERM
 
+_tst_set_oom_score_adj()
+{
+	local value="$1"
+	local path="/proc/self/oom_score_adj"
+
+	if [ ! -e "$path" ]; then
+		tst_res TINFO "oom_score_adj does not exist, skipping the adjustment"
+		return
+	fi
+
+	if ! echo "$value" > "$path"; then
+		tst_res TWARN "Can't adjust score"
+	fi
+}
+
+_tst_enable_oom_protection()
+{
+	_tst_set_oom_score_adj -1000

testcases/lib/tst_test.sh export TST_USR_GID="${LTP_USR_GID:-65534}" · line 58

Quoted 10 lines
testcases/lib/tst_test.sh +10 -0
+}
+
+_tst_disable_oom_protection()
+{
+	_tst_set_oom_score_adj 0
+}
+
+_tst_write_results()
+{
+	[ "$TST_CHILD" = 1 ] || return 0

Yes, $TST_CHILD is here visible, because it's run from the child...

Quoted 18 lines
testcases/lib/tst_test.sh +14 -0
+	[ -n "$TST_RESULTS_FILE" ] || return 0
+
+	echo "$TST_PASS $TST_FAIL $TST_BROK $TST_WARN $TST_CONF $TST_COUNT" \
+		> "$TST_RESULTS_FILE"
+}
+
+_tst_read_results()
+{
+	[ -s "$TST_RESULTS_FILE" ] || return 0
+
+	read TST_PASS TST_FAIL TST_BROK TST_WARN TST_CONF TST_COUNT \
+		< "$TST_RESULTS_FILE"
+}
+
 _tst_do_cleanup()
 {
 	if [ -n "$TST_DO_CLEANUP" -a -n "$TST_CLEANUP" -a -z "$LTP_NO_CLEANUP" ]; then

testcases/lib/tst_test.sh _tst_do_exit() · line 96

Quoted 7 lines
testcases/lib/tst_test.sh +4 -0
@@ -48,6 +90,15 @@ _tst_do_exit()
 
 	_tst_do_cleanup
 
+	# When running as the unprotected test child, only propagate the
+	# results back to the protected harness which does the teardown and
+	# prints the summary.
+	if [ "$TST_CHILD" = 1 ]; then

... but as agent noted $TST_CHILD is not visible here in _tst_do_exit() because that's run from parent shell => nack.

Quoted 13 lines
testcases/lib/tst_test.sh +5 -1
+		_tst_cleanup_timer
+		_tst_write_results
+		exit 0
+	fi
+
 	cd "$LTPROOT"
 	[ "$TST_MOUNT_FLAG" = 1 ] && tst_umount
 
@@ -788,10 +839,34 @@ tst_run()
 
 	TST_MNTPOINT="${TST_MNTPOINT:-$PWD/mntpoint}"
 
-	if [ "$TST_ALL_FILESYSTEMS" = 1 ]; then

testcases/lib/tst_test.sh tst_run() · line 848

Quoted 10 lines
testcases/lib/tst_test.sh +7 -3
-		_tst_run_tcases_per_fs
-	else
-		_tst_run_iterations
+	# Protect the harness so it survives memory pressure and can
+	# still report results, then run the test body in a child that
+	# resets OOM protection. This way workloads forked by the test
+	# remain killable, mirroring the C test harness.
+	_tst_enable_oom_protection
+
+	TST_RESULTS_FILE=$(mktemp "${TMPDIR:-/tmp}/ltp_${TST_ID}_res.XXXXXX")

Hm, writing temporary file directly to TMPDIR is not optimal, but if that was the only problem I'd be ok with it.

testcases/lib/tst_test.sh tst_run() · line 851

Quoted 2 lines
testcases/lib/tst_test.sh +2 -0
+	(
+		TST_CHILD=1

Agent is correct, this is not visible in the parent shell => I doubt it will work.

Also this is supposed to fix memcg stress tests, but it touches all shell test => very effective way to broke many tests in single commit with great change to get it quickly reverted :).

IMHO we should really rewrite the tests which does not work in tst_test.sh API into C API (better) or to the shell loader. Further complicate tst_test.sh is a way to hell.

Kind regards,
Petr

Quoted 19 lines
testcases/lib/tst_test.sh +18 -0
+		_tst_disable_oom_protection
+		trap '_tst_cleanup_timer; _tst_write_results' EXIT
+
+		if [ "$TST_ALL_FILESYSTEMS" = 1 ]; then
+			_tst_run_tcases_per_fs
+		else
+			_tst_run_iterations
+		fi
+	) &
+
+	wait $!
+	ret=$?
+
+	_tst_read_results
+	rm -f "$TST_RESULTS_FILE"
+
+	if [ $ret -gt 128 ]; then
+		tst_res TBROK "Test killed by signal $((ret - 128))!"
 	fi
Signature · 1 line
Mailing list info: https://lists.linux.it/listinfo/ltp
#4 lore raw
In reply to Andrea Cervesato

Hi Andrea, Petr,

After looking into it more, now I slightly think doing OOM protection in the shell harness isn't worth it: the benefit doesn't justify the complexity. Forking the whole test body means dealing with cleanup and timer ownership across the fork, plus passing results back from the child, and all of that would need revalidating for every shell test that goes through tst_test.sh.

Given that only a few tests actually produce memory pressure, I'd rather keep it simple and handle it locally in those tests instead of touching the common harness. Something like a small helper the test can wrap the memory hungry workload with:

    tst_oom_run()
    {
        ( echo 0 > /proc/self/oom_score_adj; exec "$@" )
    }

The harness (or the setup) protects itself with oom_score_adj -1000, and only the workload started via tst_oom_run stays killable. This keeps the harness alive to report results without forking the whole test body, and matches the C harness model where the worker is the unprotected part.

Does this direction make sense to you, or do you see a case that really needs the protection to be harness-wide?

Quoted 1 line

This email address is no longer in use :).

Signature · 5 lines
Regards,
Li Wang

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp
#5 lore raw

Hi Li, Andrea,

Quoted 8 lines

Hi Andrea, Petr,

After looking into it more, now I slightly think doing OOM protection in the shell harness isn't worth it: the benefit doesn't justify the complexity. Forking the whole test body means dealing with cleanup and timer ownership across the fork, plus passing results back from the child, and all of that would need revalidating for every shell test that goes through tst_test.sh.

+1

Quoted 9 lines

Given that only a few tests actually produce memory pressure, I'd rather keep it simple and handle it locally in those tests instead of touching the common harness. Something like a small helper the test can wrap the memory hungry workload with:

    tst_oom_run()
    {
        ( echo 0 > /proc/self/oom_score_adj; exec "$@" )
    }

+1

Maybe test should declare through TST_* variable that it needs OOM killer (approach on Andrea's v2 [1]). I'm sorry Andrea, it takes time to find a right solution.

[1] https://lore.kernel.org/ltp/[email protected]/

And again, IMHO the real solution would be to rewrite tests into C API or shell loader API.

Quoted 5 lines

The harness (or the setup) protects itself with oom_score_adj -1000, and only the workload started via tst_oom_run stays killable. This keeps the harness alive to report results without forking the whole test body, and matches the C harness model where the worker is the unprotected part.

Yes, forking (the subshell) part looked to me fragile.

Quoted 6 lines

Does this direction make sense to you, or do you see a case that really needs the protection to be harness-wide?

Quoted 1 line

This email address is no longer in use :).

+1

Kind regards,
Petr

Signature · 1 line
Mailing list info: https://lists.linux.it/listinfo/ltp
#6 lore raw
In reply to Li Wang

Hi Li,

Quoted 19 lines

Hi Andrea, Petr,

After looking into it more, now I slightly think doing OOM protection in the shell harness isn't worth it: the benefit doesn't justify the complexity. Forking the whole test body means dealing with cleanup and timer ownership across the fork, plus passing results back from the child, and all of that would need revalidating for every shell test that goes through tst_test.sh.

Given that only a few tests actually produce memory pressure, I'd rather keep it simple and handle it locally in those tests instead of touching the common harness. Something like a small helper the test can wrap the memory hungry workload with:

    tst_oom_run()
    {
        ( echo 0 > /proc/self/oom_score_adj; exec "$@" )
    }
Quoted 8 lines

The harness (or the setup) protects itself with oom_score_adj -1000, and only the workload started via tst_oom_run stays killable. This keeps the harness alive to report results without forking the whole test body, and matches the C harness model where the worker is the unprotected part.

Does this direction make sense to you, or do you see a case that really needs the protection to be harness-wide?

I think you are right, perhaps these shell tests need to be rewritten in C anyway. Most of them are unstable and fificult to maintain (like all shell scripts anyway).

I can go back to my original idea of setting oom protection only for specific tests.

Quoted 4 lines
Quoted 2 lines

This email address is no longer in use :).

Ah damn.. :-D

--
Andrea Cervesato
SUSE QE Automation Engineer Linux
[email protected]

Signature · 1 line
Mailing list info: https://lists.linux.it/listinfo/ltp