Skip to contentSkip to conversation

shell test timeout handling

PATCH New Petr Vorel 3 participants 9 messages Mar 12, 2021 – Apr 7, 2021

lore
Share & tools
lore

Petr Vorel

author
#0 lore raw

cover shell test timeout handling

Hi,

changes before previous attempt [1]
* _tst_kill_test() (Cyril)
* added test timeout03.sh
* update trap message (Joerg)

NOTE: we have no way to print summary from _tst_do_exit if cleanup get stuck, but IMHO there is no simple way how to solve it (we'd have to use mapped memory to increment counter also in shell API as Cyril suggested)

[1] https://patchwork.ozlabs.org/project/ltp/patch/[email protected]/

Petr Vorel (3):
tst_test.sh: Run cleanup also after test timeout
test: Adding timeout03.sh for testing _tst_kill_test()
zram: Increase timeout according to used devices

 lib/newlib_tests/shell/timeout03.sh           | 47 +++++++++++++++++++
 .../kernel/device-drivers/zram/zram_lib.sh    |  2 +
 testcases/lib/tst_test.sh                     | 25 ++++++++--
 3 files changed, 71 insertions(+), 3 deletions(-)
 create mode 100755 lib/newlib_tests/shell/timeout03.sh
Signature · 1 line
2.30.1
author
#1 lore raw

PATCH 1/3 tst_test.sh: Run cleanup also after test timeout

Also timeout requires to run a test cleanup (e.g. zram01.sh). Thus send first SIGINT, but keep also SIGKILL for safety reasons (after 5 sec to give some time to the cleanup function and _tst_check_security_modules()).

---
 testcases/lib/tst_test.sh | 25 ++++++++++++++++++++++---
 1 file changed, 22 insertions(+), 3 deletions(-)
testcases/lib/tst_test.sh +22 -3
index 58056e28b..1aa940f61 100644
@@ -21,7 +21,7 @@ export TST_LIB_LOADED=1
 . tst_security.sh
 
 # default trap function
-trap "tst_brk TBROK 'test interrupted'" INT
+trap "tst_brk TBROK 'test interrupted or timed out'" INT
 
 _tst_cleanup_timer()
 {
@@ -442,6 +442,26 @@ _tst_multiply_timeout()
 	return 0
 }
 
+_tst_kill_test()
+{
+	local i=10
+
+	tst_res TBROK "Test timeouted, sending SIGTERM! If you are running on slow machine, try exporting LTP_TIMEOUT_MUL > 1"
+	kill -INT -$pid
+	tst_sleep 100ms
+
+	while kill -0 $pid 2>&1 > /dev/null && [ $i -gt 0 ]; do
+		tst_res TINFO "Test is still running, waiting ${i}s"
+		sleep 1
+		i=$((i-1))
+	done
+
+	if kill -0 $pid 2>&1 > /dev/null; then
+		tst_res TBROK "Test still running, sending SIGKILL"
+		kill -KILL -$pid
+	fi
+}
+
 _tst_setup_timer()
 {
 	TST_TIMEOUT=${TST_TIMEOUT:-300}
@@ -465,8 +485,7 @@ _tst_setup_timer()
 	tst_res TINFO "timeout per run is ${h}h ${m}m ${s}s"
 
 	_tst_cleanup_timer
-
-	sleep $sec && tst_res TBROK "test killed, timeout! If you are running on slow machine, try exporting LTP_TIMEOUT_MUL > 1" && kill -9 -$pid &
+	sleep $sec && _tst_kill_test &
 
 	_tst_setup_timer_pid=$!
 }
Signature · 1 line
2.30.1
#2 lore raw

PATCH 1/3 tst_test.sh: Run cleanup also after test timeout

Hi!

testcases/lib/tst_test.sh _tst_multiply_timeout() · line 449

Quoted 11 lines

_tst_cleanup_timer()
{

testcases/lib/tst_test.sh +5 -0
@@ -442,6 +442,26 @@ _tst_multiply_timeout()
 	return 0
 }
 
+_tst_kill_test()
+{
+	local i=10
+
+	tst_res TBROK "Test timeouted, sending SIGTERM! If you are running on slow machine, try exporting LTP_TIMEOUT_MUL > 1"
                                                   ^
						   SIGINT?

testcases/lib/tst_test.sh _tst_multiply_timeout() · line 451

Quoted 2 lines
testcases/lib/tst_test.sh +2 -0
+	kill -INT -$pid
+	tst_sleep 100ms

Other than that it looks good,

Signature · 2 lines
Cyril Hrubis
[email protected]
author
#3 lore raw

PATCH 1/3 tst_test.sh: Run cleanup also after test timeout

Quoted 7 lines

Hi!

Quoted 5 lines

_tst_cleanup_timer()
{

testcases/lib/tst_test.sh +0 -0
@@ -442,6 +442,26 @@ _tst_multiply_timeout()
 	return 0
 }
Quoted 7 lines

testcases/lib/tst_test.sh _tst_multiply_timeout() · line 449

Quoted 5 lines
testcases/lib/tst_test.sh +5 -0
+_tst_kill_test()
+{
+	local i=10
+
+	tst_res TBROK "Test timeouted, sending SIGTERM! If you are running on slow machine, try exporting LTP_TIMEOUT_MUL > 1"
                                                   ^
						   SIGINT?

Oops, thanks for noticing this.

Kind regards,
Petr

Quoted 6 lines

testcases/lib/tst_test.sh _tst_multiply_timeout() · line 451

Quoted 2 lines
testcases/lib/tst_test.sh +2 -0
+	kill -INT -$pid
+	tst_sleep 100ms

Other than that it looks good,

author
#4 lore raw

PATCH 2/3 test: Adding timeout03.sh for testing _tst_kill_test()

In reply to Petr Vorel

---
new commit in this attempt

 lib/newlib_tests/shell/timeout03.sh | 47 +++++++++++++++++++++++++++++
 1 file changed, 47 insertions(+)
 create mode 100755 lib/newlib_tests/shell/timeout03.sh
lib/newlib_tests/shell/timeout03.sh new file +47 -0
new file mode 100755
index 000000000..89a4928b2
@@ -0,0 +1,47 @@
+#!/bin/sh
+# SPDX-License-Identifier: GPL-2.0-or-later
+# Copyright (c) 2021 Petr Vorel <[email protected]>
+
+# testing shell timeout handling in _tst_kill_test()
+# expected output:
+# timeout03 1 TINFO: timeout per run is 0h 0m 1s
+# timeout03 1 TINFO: testing killing test after TST_TIMEOUT
+# timeout03 1 TBROK: Test timeouted, sending SIGTERM! If you are running on slow machine, try exporting LTP_TIMEOUT_MUL > 1
+# timeout03 1 TBROK: test interrupted or timed out
+# timeout03 1 TPASS: test run cleanup after timeout
+# timeout03 1 TINFO: Test is still running, waiting 10s
+# timeout03 1 TINFO: Test is still running, waiting 9s
+# timeout03 1 TINFO: Test is still running, waiting 8s
+# timeout03 1 TINFO: Test is still running, waiting 7s
+# timeout03 1 TINFO: Test is still running, waiting 6s
+# timeout03 1 TINFO: Test is still running, waiting 5s
+# timeout03 1 TINFO: Test is still running, waiting 4s
+# timeout03 1 TINFO: Test is still running, waiting 3s
+# timeout03 1 TINFO: Test is still running, waiting 2s
+# timeout03 1 TINFO: Test is still running, waiting 1s
+# timeout03 1 TBROK: Test still running, sending SIGKILL
+# Killed
+
+TST_TESTFUNC=do_test
+TST_CLEANUP=cleanup
+
+TST_TIMEOUT=1
+. tst_test.sh
+
+do_test()
+{
+	tst_res TINFO "testing killing test after TST_TIMEOUT"
+
+	tst_sleep 2
+	tst_res TFAIL "test: running after TST_TIMEOUT"
+}
+
+cleanup()
+{
+	tst_res TPASS "test run cleanup after timeout"
+
+	tst_sleep 15 # must be higher than wait time in _tst_kill_test
+	tst_res TFAIL "cleanup: running after TST_TIMEOUT"
+}
+
+tst_run
Signature · 1 line
2.30.1
#5 lore raw

PATCH 2/3 test: Adding timeout03.sh for testing _tst_kill_test()

Hi!
Looks obviously fine, but we have to update the expected output if we are going to change the SIGTERM to SIGINT in the library.

Apart from that Reviewed-by: Cyril Hrubis <[email protected]>

Signature · 2 lines
Cyril Hrubis
[email protected]
author
#6 lore raw

PATCH 2/3 test: Adding timeout03.sh for testing _tst_kill_test()

Hi Cyril, Li,

Quoted 5 lines

Hi!
Looks obviously fine, but we have to update the expected output if we are going to change the SIGTERM to SIGINT in the library.

Apart from that Reviewed-by: Cyril Hrubis <[email protected]>

Both errors fixed and patchset merged.

Thanks a lot both for your review!

Kind regards,
Petr

author
#7 lore raw

PATCH 3/3 zram: Increase timeout according to used devices

In reply to Petr Vorel

to avoid unexpected timeout, which occurred even on just 4 zram devices.

On my system where run with ext{2,3,4}, xfs, btrfs, vfat, exfat, ntfs it run for 12 min, i.e. mean 90s. Multiply by security constant 5, expecting 450 sec for each filesystem.

---
The same as previous
https://patchwork.ozlabs.org/project/ltp/patch/[email protected]/

 testcases/kernel/device-drivers/zram/zram_lib.sh | 2 ++
 1 file changed, 2 insertions(+)
testcases/kernel/device-drivers/zram/zram_lib.sh +2 -0
index 65e431e86..fe9c915c3 100755
@@ -47,6 +47,8 @@ zram_load()
 		tst_brk TBROK "dev_num must be > 0"
 	fi
 
+	tst_set_timeout $((dev_num*450))
+
 	tst_res TINFO "create '$dev_num' zram device(s)"
 
 	modprobe zram num_devices=$dev_num || \
Signature · 1 line
2.30.1
#8 lore raw

cover shell test timeout handling

In reply to Petr Vorel
Quoted 11 lines · Petr Vorel

Hi,

changes before previous attempt [1]
* _tst_kill_test() (Cyril)
* added test timeout03.sh
* update trap message (Joerg)

NOTE: we have no way to print summary from _tst_do_exit if cleanup get stuck, but IMHO there is no simple way how to solve it (we'd have to use mapped memory to increment counter also in shell API as Cyril suggested)

Quoted 8 lines

[1]
https://patchwork.ozlabs.org/project/ltp/patch/[email protected]/

Petr Vorel (3):
tst_test.sh: Run cleanup also after test timeout
test: Adding timeout03.sh for testing _tst_kill_test()
zram: Increase timeout according to used devices

Nice work!

For series:

Signature · 5 lines
Regards,
Li Wang
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linux.it/pipermail/ltp/attachments/20210315/d14a3ccf/attachment.htm>