Skip to contentSkip to conversation

shell: Fix orphan timeout sleep processes

PATCH New Joerg Vehlow 3 participants 8 messages May 7, 2021 – May 8, 2021

lore
Share & tools
lore
#0 lore raw

v2 cover shell: Fix orphan timeout sleep processes

Hi,

Updates to v1: Instead of a c based solution this uses the shell based fix suggested by Li

J?rg

#1 lore raw

PATCH v2 1/2 shell: Extend timeout tests

From: Joerg Vehlow <[email protected]>

- Add test, that verifies, that the timeout is working as expected.
- Add test case to check if there are processes left, after a test finished without running into a timeout.
- Add execution of timeout03.sh and verification of return code.

---
 lib/newlib_tests/shell/test_timeout.sh | 178 ++++++++++++++++++++++---
 lib/newlib_tests/shell/timeout04.sh    |  22 +++
 2 files changed, 179 insertions(+), 21 deletions(-)
 create mode 100755 lib/newlib_tests/shell/timeout04.sh
lib/newlib_tests/shell/test_timeout.sh +157 -21
index 26477d6f2..06f3acf88 100755
@@ -1,42 +1,178 @@
 #!/bin/sh
 # SPDX-License-Identifier: GPL-2.0-or-later
 # Copyright (c) 2019 Petr Vorel <[email protected]>
+# Copyright (c) 2021 Joerg Vehlow <[email protected]>
 
 PATH="$(dirname $0):$(dirname $0)/../../../testcases/lib/:$PATH"
 
+# Test cases are separated by newlines.
+# Every test has the following fields in this order:
+# file
+# timeout_mul
+# use_cat
+# max_runtime
+# expected_exit_code
+# expected passes
+# expected failed
+# expected broken
+# description
+# Whole lines can be commented out using "#"
 DATA="
-timeout01.sh||0
-timeout02.sh||0
-timeout02.sh|foo|2
-timeout02.sh|2|0
-timeout01.sh|2|0
-timeout02.sh|1.1|0
-timeout02.sh|-10|2
-timeout02.sh|-0.1|0
-timeout02.sh|-1.1|2
-timeout02.sh|-10.1|2
+timeout01.sh|     |0|  |0
+timeout02.sh|     |0|  |0
+timeout02.sh|  foo|0|  |2
+timeout02.sh|    2|0|  |0
+timeout01.sh|    2|0|  |0
+timeout02.sh|  1.1|0|  |0
+timeout02.sh|  -10|0|  |2
+timeout02.sh| -0.1|0|  |0
+timeout02.sh| -1.1|0|  |2
+timeout02.sh|-10.1|0|  |2
+timeout03.sh|     |0|12|137| | | |Test kill if test does not terminate by SIGINT
+timeout04.sh|     |0|  |  2|0|0|1|Verify that timeout is enforced
+timeout02.sh|    2|1| 2|   |1|0|0|Test termination of timeout process
 "
 
+# Executes a test
+# Parameter:
+#  - test:    The test to execute
+#  - timeout: The timeout multiplicator (optional)
+#  - use_cat: Pipe the output of the command through cat (optional)
+#             If this is used, the exit code is NOT returned!
+#
+# The function returns the following global variables:
+# - test_exit:     The exit code of the test
+# - test_duration: The duration of the test in seconds
+# - test_output:   The full output of the test
+# - test_passed:   The number of passed tests parsed from the summary
+# - test_failed:   The number of failed tests parsed from the summary
+# - test_broken:   The number of broken tests parsed from the summary
+run_test()
+{
+	local test=$1
+	local timeout=$2
+	local use_cat=$3
+	local tmpfile start end;
+
+	tmpfile=$(mktemp -t ltp_timeout_XXXXXXXX)
+	start=$(date +%s)
+	# We have to use set monitor mode (set -m) here.
+	# In most shells in most cases this creates a
+	# new process group for the next command.
+	# A process group is required for the timeout functionality,
+	# because it sends signals to the whole process group.
+	set -m
+	# The use_cat is for testing if any programm using stdout
+	# is still running, after the test finished.
+	# cat will wait for its stdin to be closed.
+	#
+	# In the pure shell implementation of the timeout handling,
+	# the sleep process was never killed, when a test finished
+	# before the timeout.
+	if [ "$use_cat" = "1" ]; then
+		LTP_TIMEOUT_MUL=$timeout $test 2>&1 | cat >$tmpfile
+	else
+		LTP_TIMEOUT_MUL=$timeout $test 1>$tmpfile 2>&1
+	fi
+	test_exit=$?
+	set +m
+	end=$(date +%s)
+
+	test_duration=$((end - start))
+	test_output=$(cat $tmpfile)
+	rm $tmpfile
+
+	eval $(echo "$test_output" | awk '
+		BEGIN {sum=0} 
+		$1 == "Summary:" {
+			sum=1;
+		}
+		sum && ( \
+			   $1 == "passed" \
+			|| $1 == "failed" \
+			|| $1 == "broken") {
+			print "test_" $1 "=" $2
+		}
+	')
+}
+
 echo "Testing timeout in shell API"
 echo
 
 failed=0
-for i in $DATA; do
-	file=$(echo $i | cut -d'|' -f1)
-	timeout=$(echo $i | cut -d'|' -f2)
-	exp_exit=$(echo $i | cut -d'|' -f3)
-
-	echo "=== $file (LTP_TIMEOUT_MUL='$timeout') ==="
-	LTP_TIMEOUT_MUL=$timeout $file
-	ret=$?
-	if [ $ret -ne $exp_exit ]; then
-		echo "FAILED (exit code: $ret, expected $exp_exit)"
-		failed=$((failed+1))
+test_nr=0
+
+old_ifs="$IFS"
+IFS=$(printf "\n\b")
+
+# Remove comments and empty lines
+CLEANED_DATA=$(echo "$DATA" | sed '/^\s*#/d;/^\s*$/d')
+test_max=$(echo "$CLEANED_DATA" | wc -l)
+for d in $CLEANED_DATA; do
+	IFS="$old_ifs"
+
+	file=$(echo $d | cut -d'|' -f1 | xargs)
+	timeout=$(echo $d | cut -d'|' -f2 | xargs)
+	use_cat=$(echo $d | cut -d'|' -f3 | xargs)
+	max_runtime=$(echo $d | cut -d'|' -f4 | xargs)
+	max_runtime=${max_runtime:--1}
+	exp_exit=$(echo $d | cut -d'|' -f5 | xargs)
+	exp_exit=${exp_exit:--1}
+	exp_passed=$(echo $d | cut -d'|' -f6 | xargs)
+	exp_passed=${exp_passed:--1}
+	exp_failed=$(echo $d | cut -d'|' -f7 | xargs)
+	exp_failed=${exp_failed:--1}
+	exp_broken=$(echo $d | cut -d'|' -f8 | xargs)
+	exp_broken=${exp_broken:--1}
+	description=$(echo $d | cut -d'|' -f9)
+
+	test_nr=$(($test_nr + 1))
+
+	cur_fails=0
+
+	if [ -z "$description" ]; then
+		description="$file (LTP_TIMEOUT_MUL='$timeout')"
+	fi
+
+	echo "=== $test_nr/$test_max $description ==="
+	run_test "$file" "$timeout" "$use_cat"
+
+	if [ $max_runtime -ne -1 ] && [ $test_duration -gt $max_runtime ]; then
+		echo "FAILED (runtime: $test_duration, expected less than $max_runtime)"
+		cur_fails=$((cur_fails + 1))
+	fi
+
+	if [ $exp_passed -ne -1 ] && [ $exp_passed -ne $test_passed ]; then
+		echo "FAILED (passes: $test_passed, expected $exp_passed)"
+		cur_fails=$((cur_fails + 1))
+	fi
+
+	if [ $exp_failed -ne -1 ] && [ $exp_failed -ne $test_failed ]; then
+		echo "FAILED (failed: $test_failed, expected $exp_failed)"
+		cur_fails=$((cur_fails + 1))
+	fi
+
+	if [ $exp_broken -ne -1 ] && [ $exp_broken -ne $test_broken ]; then
+		echo "FAILED (broken: $test_broken, expected $exp_broken)"
+		cur_fails=$((cur_fails + 1))
+	fi
+
+	if [ $exp_exit -ne -1 ] && [ $test_exit -ne $exp_exit ]; then
+		echo "FAILED (exit code: $test_exit, expected $exp_exit)"
+		cur_fails=$((cur_fails + 1))
+	fi
+
+	if [ $cur_fails -gt 0 ]; then
+		failed=$((failed + 1))
+		echo "--------"
+		echo "$test_output"
+		echo "--------"
 	else
 		echo "PASSED"
 	fi
 	echo
 done
+IFS="$old_ifs"
 
 echo "Failed tests: $failed"
 exit $failed
lib/newlib_tests/shell/timeout04.sh new file +22 -0
new file mode 100755
index 000000000..0a6ba053c
@@ -0,0 +1,22 @@
+#!/bin/sh
+# SPDX-License-Identifier: GPL-2.0-or-later
+# Copyright (c) 2021 Joerg Vehlow <[email protected]>
+
+TST_TESTFUNC=do_test
+
+TST_TIMEOUT=1
+. tst_test.sh
+
+do_test()
+{
+	tst_res TINFO "Start"
+    sleep 5
+    tst_res TFAIL "End"
+}
+
+do_cleanup()
+{
+    tst_res TINFO "cleanup"
+}
+
+tst_run
Signature · 1 line
2.25.1
#2 lore raw

PATCH v2 2/2 shell: Prevent orphan timeout sleep processes

In reply to Joerg Vehlow

From: Joerg Vehlow <[email protected]>

The implementation of the timeout handling failed to terminate the sleep process, because the command "sleep $sec && _tst_kill_test &" spawns two
processes: A shell, that is used to execute the sleep and after sleep terminates it executes _tst_kill_test. The pid stored in $! is the pid of the shell process.
When the test finished, the timeout process is supposed to be killed, but only the shell process is killed, not the sleep. If the output of the test is piped somewhere else, the pipe stays open, until the sleep finished, because it still has stdout open.
Additionally, if a lot of short tests with high timeouts are executed, a lot of sleep processes will linger around.

---
 testcases/lib/tst_test.sh | 30 +++++++++++++++++++++---------
 1 file changed, 21 insertions(+), 9 deletions(-)
testcases/lib/tst_test.sh +21 -9
index 951518785..7ceddff04 100644
@@ -23,14 +23,6 @@ export TST_LIB_LOADED=1
 # default trap function
 trap "tst_brk TBROK 'test interrupted or timed out'" INT
 
-_tst_cleanup_timer()
-{
-	if [ -n "$_tst_setup_timer_pid" ]; then
-		kill $_tst_setup_timer_pid 2>/dev/null
-		wait $_tst_setup_timer_pid 2>/dev/null
-	fi
-}
-
 _tst_do_exit()
 {
 	local ret=0
@@ -463,6 +455,25 @@ _tst_kill_test()
 	fi
 }
 
+_tst_cleanup_timer()
+{
+	if [ -n "$_tst_setup_timer_pid" ]; then
+		kill -TERM $_tst_setup_timer_pid 2>/dev/null
+		wait $_tst_setup_timer_pid 2>/dev/null
+	fi
+}
+
+_tst_timeout_process()
+{
+	local sleep_pid
+
+	sleep $sec &
+	sleep_pid=$!
+	#trap "kill $sleep_pid; exit" TERM
+	wait $sleep_pid
+	_tst_kill_test
+}
+
 _tst_setup_timer()
 {
 	TST_TIMEOUT=${TST_TIMEOUT:-300}
@@ -486,7 +497,8 @@ _tst_setup_timer()
 	tst_res TINFO "timeout per run is ${h}h ${m}m ${s}s"
 
 	_tst_cleanup_timer
-	sleep $sec && _tst_kill_test &
+
+	_tst_timeout_process &
 
 	_tst_setup_timer_pid=$!
 }
Signature · 1 line
2.25.1
#3 lore raw

PATCH v2 2/2 shell: Prevent orphan timeout sleep processes

Hi Joerg,

The patchset looks good except for few minor issues, we can help fix them when merging.

lib/newlib_tests/shell/timeout03.sh +2 -2
@@ -32,7 +32,7 @@ do_test()
 {
        tst_res TINFO "testing killing test after TST_TIMEOUT"
 
-       tst_sleep 2
+       sleep 2
        tst_res TFAIL "test: running after TST_TIMEOUT"
 }
 
@@ -40,7 +40,7 @@ cleanup()
 {
        tst_res TPASS "test run cleanup after timeout"
 
-       tst_sleep 15 # must be higher than wait time in _tst_kill_test
+       sleep 15 # must be higher than wait time in _tst_kill_test
        tst_res TFAIL "cleanup: running after TST_TIMEOUT"
 }
 
(quoted diff) +0 -0

b/lib/newlib_tests/shell/timeout04.sh
index 0a6ba053c..c702905f3 100755

lib/newlib_tests/shell/timeout04.sh +1 -1
@@ -9,7 +9,7 @@ TST_TIMEOUT=1
 
 do_test()
 {
-       tst_res TINFO "Start"
+    tst_res TINFO "Start"
     sleep 5
     tst_res TFAIL "End"
 }
testcases/lib/tst_test.sh +1 -1
index 7ceddff04..ed2699175 100644
@@ -469,7 +469,7 @@ _tst_timeout_process()
 
        sleep $sec &
        sleep_pid=$!
-       #trap "kill $sleep_pid; exit" TERM
+       trap "kill $sleep_pid; exit" TERM
        wait $sleep_pid
        _tst_kill_test
 }

================
(This below is not related to your patches)

But there is another issue I found that the timeout03 can NOT be killed after timed out in calling cleanup(), the reason is tst_brk stop the _tst_kill_test running in the background so that it does not output as what we expected:

# timeout03 1 TINFO: timeout per run is 0h 0m 1s
# timeout03 1 TINFO: testing killing test after TST_TIMEOUT
# timeout03 1 TBROK: Test timeouted, sending SIGINT! If you are
running on slow machine, try exporting LTP_TIMEOUT_MUL > 1
# timeout03 1 TBROK: test interrupted or timed out
# timeout03 1 TPASS: test run cleanup after timeout
# timeout03 1 TINFO: Test is still running, waiting 10s
# timeout03 1 TINFO: Test is still running, waiting 9s
# timeout03 1 TINFO: Test is still running, waiting 8s
# timeout03 1 TINFO: Test is still running, waiting 7s
# timeout03 1 TINFO: Test is still running, waiting 6s
# timeout03 1 TINFO: Test is still running, waiting 5s
# timeout03 1 TINFO: Test is still running, waiting 4s
# timeout03 1 TINFO: Test is still running, waiting 3s
# timeout03 1 TINFO: Test is still running, waiting 2s
# timeout03 1 TINFO: Test is still running, waiting 1s
# timeout03 1 TBROK: Test still running, sending SIGKILL
# Killed

--
Regards,
Li Wang

#4 lore raw

PATCH v2 2/2 shell: Prevent orphan timeout sleep processes

Hi Li, Joerg,

thanks both for your work!

lib/newlib_tests/shell/timeout03.sh do_test() · line 35

Quoted 7 lines

+++ b/lib/newlib_tests/shell/timeout03.sh

lib/newlib_tests/shell/timeout03.sh +1 -1
@@ -32,7 +32,7 @@ do_test()
 {
        tst_res TINFO "testing killing test after TST_TIMEOUT"
 
-       tst_sleep 2
+       sleep 2

+1
Although tst_test.sh has many LTP custom binary dependencies I agree to use simple sleep for seconds.

lib/newlib_tests/shell/timeout03.sh cleanup() · line 43

Quoted 9 lines
lib/newlib_tests/shell/timeout03.sh +1 -1
        tst_res TFAIL "test: running after TST_TIMEOUT"
 }
 
@@ -40,7 +40,7 @@ cleanup()
 {
        tst_res TPASS "test run cleanup after timeout"
 
-       tst_sleep 15 # must be higher than wait time in _tst_kill_test
+       sleep 15 # must be higher than wait time in _tst_kill_test

and here.

Quoted 22 lines
lib/newlib_tests/shell/timeout03.sh +0 -0
        tst_res TFAIL "cleanup: running after TST_TIMEOUT"
 }
 
lib/newlib_tests/shell/timeout04.sh +1 -1
index 0a6ba053c..c702905f3 100755
@@ -9,7 +9,7 @@ TST_TIMEOUT=1
 
 do_test()
 {
-       tst_res TINFO "Start"
+    tst_res TINFO "Start"
     sleep 5
     tst_res TFAIL "End"
 }
testcases/lib/tst_test.sh +0 -0
index 7ceddff04..ed2699175 100644

testcases/lib/tst_test.sh _tst_timeout_process() · line 472

Quoted 5 lines
testcases/lib/tst_test.sh +1 -1
@@ -469,7 +469,7 @@ _tst_timeout_process()
 
        sleep $sec &
        sleep_pid=$!
-       #trap "kill $sleep_pid; exit" TERM
+       trap "kill $sleep_pid; exit" TERM

I guess trap use is needed, and commented out for testing.

Quoted 5 lines
testcases/lib/tst_test.sh +0 -0
        wait $sleep_pid
        _tst_kill_test
 }
Quoted 7 lines

================
(This below is not related to your patches)

But there is another issue I found that the timeout03 can NOT be killed after timed out in calling cleanup(), the reason is tst_brk stop the _tst_kill_test running in the background so that it does not output as what we expected:

Good catch, I'll try to have a look as well.

Kind regards,
Petr

Quoted 18 lines
# timeout03 1 TINFO: timeout per run is 0h 0m 1s
# timeout03 1 TINFO: testing killing test after TST_TIMEOUT
# timeout03 1 TBROK: Test timeouted, sending SIGINT! If you are
running on slow machine, try exporting LTP_TIMEOUT_MUL > 1
# timeout03 1 TBROK: test interrupted or timed out
# timeout03 1 TPASS: test run cleanup after timeout
# timeout03 1 TINFO: Test is still running, waiting 10s
# timeout03 1 TINFO: Test is still running, waiting 9s
# timeout03 1 TINFO: Test is still running, waiting 8s
# timeout03 1 TINFO: Test is still running, waiting 7s
# timeout03 1 TINFO: Test is still running, waiting 6s
# timeout03 1 TINFO: Test is still running, waiting 5s
# timeout03 1 TINFO: Test is still running, waiting 4s
# timeout03 1 TINFO: Test is still running, waiting 3s
# timeout03 1 TINFO: Test is still running, waiting 2s
# timeout03 1 TINFO: Test is still running, waiting 1s
# timeout03 1 TBROK: Test still running, sending SIGKILL
# Killed
#5 lore raw

PATCH v2 2/2 shell: Prevent orphan timeout sleep processes

Quoted 8 lines
Quoted 7 lines

================
(This below is not related to your patches)

But there is another issue I found that the timeout03 can NOT be killed after timed out in calling cleanup(), the reason is tst_brk stop the _tst_kill_test running in the background so that it does not output as what we expected:

Good catch, I'll try to have a look as well.

The simplest way I can think of is to let _tst_kill_test ignores the TERM and INT signals. If you agree to this, we can fix it in a separate patch:).

testcases/lib/tst_test.sh +2 -0
@@ -439,6 +439,8 @@ _tst_kill_test()
 {
        local i=10
 
+       trap " " TERM INT
+
        tst_res TBROK "Test timeouted, sending SIGINT! If you are

running on slow machine, try exporting LTP_TIMEOUT_MUL > 1"
kill -INT -$pid
tst_sleep 100ms

Signature · 2 lines
Regards,
Li Wang
#6 lore raw

PATCH v2 2/2 shell: Prevent orphan timeout sleep processes

Hi Li,

Quoted 13 lines
Quoted 5 lines
Quoted 4 lines

But there is another issue I found that the timeout03 can NOT be killed after timed out in calling cleanup(), the reason is tst_brk stop the _tst_kill_test running in the background so that it does not output as what we expected:

Good catch, I'll try to have a look as well.

The simplest way I can think of is to let _tst_kill_test ignores the TERM and INT signals. If you agree to this, we can fix it in a separate patch:).

testcases/lib/tst_test.sh _tst_kill_test() · line 442

Quoted 4 lines
testcases/lib/tst_test.sh +1 -0
@@ -439,6 +439,8 @@ _tst_kill_test()
 {
        local i=10
 
+       trap " " TERM INT

Thanks!

BTW I'm surprised " " works, I'd expect : would have to be used.

Kind regards,
Petr

#7 lore raw

PATCH v2 2/2 shell: Prevent orphan timeout sleep processes

Quoted 7 lines

testcases/lib/tst_test.sh _tst_kill_test() · line 442

Quoted 7 lines
testcases/lib/tst_test.sh +1 -0
@@ -439,6 +439,8 @@ _tst_kill_test()
 {
        local i=10
 
+       trap " " TERM INT

Seems no need to ignore the SIGTERM, because it only sends in the early (before timeout) phase, the _tst_kill_test has no chance to catch a SIGTERM.

Quoted 1 line

Thanks for the review! I will send a patch for it.

Quoted 4 lines

Thanks!

BTW I'm surprised " " works, I'd expect : would have to be used.

" " or '' define none string makes it ignore signals.

Signature · 2 lines
Regards,
Li Wang