Terminate leftover subprocesses when main test process crashes
PATCH 1/1 Terminate leftover subprocesses when main test process crashes
When the main test process crashes or gets killed e.g. by OOM killer, the watchdog process currently does not clean up any remaining child processes. Fix this by sending SIGKILL to the test process group when the watchdog process gets notified that the main test process has exited for any reason.
Fixes #892
--- lib/newlib_tests/runtest.sh | 3 +- lib/newlib_tests/test_children_cleanup.c | 43 +++++++++++++++++++++++ lib/newlib_tests/test_children_cleanup.sh | 17 +++++++++ lib/tst_test.c | 9 +++++ 4 files changed, 71 insertions(+), 1 deletion(-) create mode 100644 lib/newlib_tests/test_children_cleanup.c create mode 100755 lib/newlib_tests/test_children_cleanup.sh
lib/newlib_tests/runtest.sh +2 -1
@@ -4,7 +4,8 @@ LTP_C_API_TESTS="${LTP_C_API_TESTS:-test05 test07 test09 test12 test15 test18 tst_needs_cmds01 tst_needs_cmds02 tst_needs_cmds03 tst_needs_cmds06 tst_needs_cmds07 tst_bool_expr test_exec test_timer tst_res_hexd tst_strstatus -tst_fuzzy_sync03 test_zero_hugepage.sh test_kconfig.sh}" +tst_fuzzy_sync03 test_zero_hugepage.sh test_kconfig.sh +test_children_cleanup.sh}" LTP_SHELL_API_TESTS="${LTP_SHELL_API_TESTS:-shell/tst_check_driver.sh shell/tst_check_kconfig0[1-5].sh shell/net/*.sh}"
lib/newlib_tests/test_children_cleanup.c new file +43 -0
43 lines not on this page · raw message
lib/newlib_tests/test_children_cleanup.sh new file +17 -0
@@ -0,0 +1,17 @@ +#!/bin/sh +# SPDX-License-Identifier: GPL-2.0-or-later +# Copyright (c) 2022 SUSE LLC <[email protected]> + +TMPFILE="/tmp/ltp_children_cleanup_$$.log" +CHILD_PID=`./test_children_cleanup 2>&1 | sed -n 's/^.*Forked child \([0-9]*\)$/\1/p'` + +if [ "x$CHILD_PID" = "x" ]; then + echo "TFAIL: Child process was not created" + exit 1 +elif ! kill -s 0 $CHILD_PID &>/dev/null; then + echo "TPASS: Child process was cleaned up" + exit 0 +else + echo "TFAIL: Child process was left behind" + exit 1 +fi
lib/tst_test.c +9 -0
@@ -1399,6 +1399,13 @@ static void sigint_handler(int sig LTP_ATTRIBUTE_UNUSED) } } +static void sigchild_handler(int sig LTP_ATTRIBUTE_UNUSED) +{ +/* If the test process is dead, send SIGKILL to its children */ +if (kill (test_pid,0 )) +kill (-test_pid,SIGKILL ); +} +unsigned int tst_timeout_remaining (void ) {static struct timespec now;
@@ -1481,6 +1488,7 @@ static int fork_testrun(void)tst_disable_oom_protection (0 );SAFE_SIGNAL (SIGALRM ,SIG_DFL );SAFE_SIGNAL (SIGUSR1 ,SIG_DFL ); +SAFE_SIGNAL (SIGCHLD ,SIG_DFL );SAFE_SIGNAL (SIGINT ,SIG_DFL );SAFE_SETPGID (0 ,0 );testrun ();
@@ -1560,6 +1568,7 @@ void tst_run_tcases(int argc, char *argv[], struct tst_test *self)SAFE_SIGNAL (SIGALRM , alarm_handler);SAFE_SIGNAL (SIGUSR1 , heartbeat_handler); +SAFE_SIGNAL (SIGCHLD , sigchild_handler);if (tst_test->test_variants) test_variants = tst_test->test_variants;
Signature · 5 lines
2.34.1 -- Mailing list info: https://lists.linux.it/listinfo/ltp
Quoted 34 lines · Martin Doucha
When the main test process crashes or gets killed e.g. by OOM killer, the watchdog process currently does not clean up any remaining child processes. Fix this by sending SIGKILL to the test process group when the watchdog process gets notified that the main test process has exited for any reason.
lib/tst_test.c +7
-0@@ -1399,6 +1399,13 @@ static void sigint_handler(int sig LTP_ATTRIBUTE_UNUSED) } } +static void sigchild_handler(int sig LTP_ATTRIBUTE_UNUSED) +{ +/* If the test process is dead, send SIGKILL to its children */ +if (kill (test_pid,0 )) +kill (-test_pid,SIGKILL ); +} +unsigned int tst_timeout_remaining (void ) {static struct timespec now;lib/tst_test.c +1
-0@@ -1481,6 +1488,7 @@ static int fork_testrun(void)tst_disable_oom_protection (0 );SAFE_SIGNAL (SIGALRM ,SIG_DFL );SAFE_SIGNAL (SIGUSR1 ,SIG_DFL ); +SAFE_SIGNAL (SIGCHLD ,SIG_DFL );SAFE_SIGNAL (SIGINT ,SIG_DFL );SAFE_SETPGID (0 ,0 );testrun ();
lib/tst_test.c void tst_run_tcases(int argc, char *argv[], struct · line 1573
Quoted 6 lines
lib/tst_test.c +1
-0@@ -1560,6 +1568,7 @@ void tst_run_tcases(int argc, char *argv[], struct tst_test *self)SAFE_SIGNAL (SIGALRM ,alarm_handler );SAFE_SIGNAL (SIGUSR1 , heartbeat_handler); +SAFE_SIGNAL (SIGCHLD , sigchild_handler);
Do we really need setup this signal handler for SIGCHILD?
Since we have already called 'SAFE_WAITPID(test_pid, &status, 0)' in the library process (lib_pid) which rely on SIGCHILD as well. And even this handler will be called everytime when test exit normally.
Maybe better just add a kill function to cleanup the remain descendants if main test process exit with abonormal status.
e.g.
lib/tst_test.c +2 -0
@@ -1503,6 +1503,8 @@ static int fork_testrun(void)if (WIFEXITED (status) &&WEXITSTATUS (status))return WEXITSTATUS (status); +kill (-test_pid,SIGKILL ); +if (WIFSIGNALED (status) &&WTERMSIG (status) ==SIGKILL ) {tst_res (TINFO ,"If you are running on slow machine, " "try exporting LTP_TIMEOUT_MUL > 1" );
Signature · 2 lines
Regards, Li Wang
Signature · 2 lines
-- Mailing list info: https://lists.linux.it/listinfo/ltp
Quoted 39 lines · Li Wang
Quoted 35 lines · Martin Doucha
When the main test process crashes or gets killed e.g. by OOM killer, the watchdog process currently does not clean up any remaining child processes. Fix this by sending SIGKILL to the test process group when the watchdog process gets notified that the main test process has exited for any reason.
lib/tst_test.c +8
-0@@ -1399,6 +1399,13 @@ static void sigint_handler(int sig LTP_ATTRIBUTE_UNUSED) } } +static void sigchild_handler(int sig LTP_ATTRIBUTE_UNUSED) +{ +/* If the test process is dead, send SIGKILL to its children */ +if (kill (test_pid,0 )) +kill (-test_pid,SIGKILL ); +} +unsigned int tst_timeout_remaining (void ) {static struct timespec now;@@ -1481,6 +1488,7 @@ static int fork_testrun(void)tst_disable_oom_protection (0 );SAFE_SIGNAL (SIGALRM ,SIG_DFL );SAFE_SIGNAL (SIGUSR1 ,SIG_DFL ); +SAFE_SIGNAL (SIGCHLD ,SIG_DFL );SAFE_SIGNAL (SIGINT ,SIG_DFL );SAFE_SETPGID (0 ,0 );testrun ();@@ -1560,6 +1568,7 @@ void tst_run_tcases(int argc, char *argv[], struct tst_test *self)
Quoted 6 lines
lib/tst_test.c void tst_run_tcases(int argc, char *argv[], struct tst_test *self) · line 1571
Quoted 3 lines
lib/tst_test.c +1
-0SAFE_SIGNAL (SIGALRM , alarm_handler);SAFE_SIGNAL (SIGUSR1 , heartbeat_handler); +SAFE_SIGNAL (SIGCHLD , sigchild_handler);Do we really need setup this signal handler for SIGCHILD?
I had same question.
Quoted 13 lines
Since we have already called '
SAFE_WAITPID(test_pid, &status, 0)' in the library process (lib_pid) which rely on SIGCHILD as well. And even this handler will be called everytime when test exit normally.Maybe better just add a kill function to cleanup the remain descendants if main test process exit with abonormal status.
e.g.
lib/tst_test.c static int fork_testrun(void) · line 1506
Quoted 4 lines
lib/tst_test.c +1
-0@@ -1503,6 +1503,8 @@ static int fork_testrun(void)if (WIFEXITED (status) &&WEXITSTATUS (status))return WEXITSTATUS (status); +kill (-test_pid,SIGKILL );
Could we skip the call if forks_child == 0 ?
Quoted 11 lines
lib/tst_test.c +1
-1+if (WIFSIGNALED (status) &&WTERMSIG (status) ==SIGKILL ) {tst_res (TINFO ,"If you are running on slow machine, " "try exporting LTP_TIMEOUT_MUL > 1" ); --Regards,
Li Wang--
Mailing list info: https://lists.linux.it/listinfo/ltp
Signature · 1 line
Mailing list info: https://lists.linux.it/listinfo/ltp
Quoted 67 lines · Jan Stancek
Quoted 13 lines · Li Wang
Quoted 9 lines · Martin Doucha
When the main test process crashes or gets killed e.g. by OOM killer, the watchdog process currently does not clean up any remaining child processes. Fix this by sending SIGKILL to the test process group when the watchdog process gets notified that the main test process has exited for any reason.
Quoted 3 lines
lib/tst_test.c static void sigint_handler(int sig
Quoted 3 lines
lib/tst_test.c +0
-0@@ -1399,6 +1399,13 @@ static void sigint_handler(int sig
LTP_ATTRIBUTE_UNUSED)Quoted 14 lines
Quoted 14 lines
lib/tst_test.c +7
-0} } +static void sigchild_handler(int sig LTP_ATTRIBUTE_UNUSED) +{ +/* If the test process is dead, send SIGKILL to its children */ +if (kill (test_pid,0 )) +kill (-test_pid,SIGKILL ); +} +unsigned int tst_timeout_remaining (void ) {static struct timespec now;@@ -1481,6 +1488,7 @@ static int fork_testrun(void)Quoted 8 lines
lib/tst_test.c void tst_run_tcases(int argc, char *argv[],
Quoted 8 lines
lib/tst_test.c +1
-0tst_disable_oom_protection (0 );SAFE_SIGNAL (SIGALRM ,SIG_DFL );SAFE_SIGNAL (SIGUSR1 ,SIG_DFL ); +SAFE_SIGNAL (SIGCHLD ,SIG_DFL );SAFE_SIGNAL (SIGINT ,SIG_DFL );SAFE_SETPGID (0 ,0 );testrun ();@@ -1560,6 +1568,7 @@ void tst_run_tcases(int argc, char *argv[],struct
tst_test*self)Quoted 7 lines
lib/tst_test.c void tst_run_tcases(int argc, char *argv[], · line 1570
Quoted 4 lines
lib/tst_test.c +1
-0SAFE_SIGNAL (SIGALRM , alarm_handler);SAFE_SIGNAL (SIGUSR1 , heartbeat_handler); +SAFE_SIGNAL (SIGCHLD , sigchild_handler);Do we really need setup this signal handler for SIGCHILD?
I had same question.
Quoted 16 lines
Since we have already called '
SAFE_WAITPID(test_pid, &status, 0)' in the library process (lib_pid) which rely on SIGCHILD as well. And even this handler will be called everytime when test exit normally.Maybe better just add a kill function to cleanup the remain descendants if main test process exit with abonormal status.
e.g.
lib/tst_test.c +0
-0@@ -1503,6 +1503,8 @@ static int fork_testrun(void)if (WIFEXITED (status) &&WEXITSTATUS (status))return WEXITSTATUS (status);
Quoted 4 lines
lib/tst_test.c static int fork_testrun(void) · line 1506
Quoted 1 line
lib/tst_test.c +1
-0+kill (-test_pid,SIGKILL );Could we skip the call if
forks_child== 0 ?
+1 Obviously yes!
Signature · 2 lines
Regards, Li Wang
Signature · 2 lines
-- Mailing list info: https://lists.linux.it/listinfo/ltp
Quoted 26 lines · Li Wang
On Fri, Feb 11, 2022 at 12:18 AM Martin Doucha <[email protected] <mailto:[email protected]>> wrote:
@@ -1560,6 +1568,7 @@ void tst_run_tcases(int argc, char *argv[], struct tst_test *self) SAFE_SIGNAL(SIGALRM, alarm_handler); SAFE_SIGNAL(SIGUSR1, heartbeat_handler); + SAFE_SIGNAL(SIGCHLD, sigchild_handler);Do we really need setup this signal handler for SIGCHILD?
Since we have already called '
SAFE_WAITPID(test_pid, &status, 0)' in the library process (lib_pid) which rely on SIGCHILD as well. And even this handler will be called everytime when test exit normally.Maybe better just add a kill function to cleanup the remain descendants if main test process exit with abonormal status.
e.g.
lib/tst_test.c +0
-0@@ -1503,6 +1503,8 @@ static int fork_testrun(void) if (WIFEXITED(status) && WEXITSTATUS(status)) return WEXITSTATUS(status);
Quoted 2 lines
+kill(-test_pid, SIGKILL);
This will not work because at this point, the child process was already destroyed by waitpid() and all its remaining children were moved under PID 1 (init). The only place where the grandchildren are still reachable this way is in SIGCHLD handler while the dead child process still exists in zombie state.
Signature · 10 lines
Martin Doucha [email protected] QA Engineer for Software Maintenance SUSE LINUX, s.r.o. CORSO IIa Krizikova 148/34 186 00 Prague 8 Czech Republic -- Mailing list info: https://lists.linux.it/listinfo/ltp
Quoted 28 lines · Martin Doucha
Quoted 27 lines · Li Wang
On Fri, Feb 11, 2022 at 12:18 AM Martin Doucha <[email protected] <mailto:[email protected]>> wrote:
@@ -1560,6 +1568,7 @@ void tst_run_tcases(int argc, char *argv[], struct tst_test *self) SAFE_SIGNAL(SIGALRM, alarm_handler); SAFE_SIGNAL(SIGUSR1, heartbeat_handler); + SAFE_SIGNAL(SIGCHLD, sigchild_handler);Do we really need setup this signal handler for SIGCHILD?
Since we have already called '
SAFE_WAITPID(test_pid, &status, 0)' in the library process (lib_pid) which rely on SIGCHILD as well. And even this handler will be called everytime when test exit normally.Maybe better just add a kill function to cleanup the remain descendants if main test process exit with abonormal status.
e.g.
lib/tst_test.c +0
-0@@ -1503,6 +1503,8 @@ static int fork_testrun(void)if (WIFEXITED (status) &&WEXITSTATUS (status))return WEXITSTATUS (status);
Quoted 4 lines
lib/tst_test.c static int fork_testrun(void) · line 1506
Quoted 1 line
lib/tst_test.c +1
-0+kill (-test_pid,SIGKILL );This will not work because at this point, the child process was already destroyed by
waitpid()and all its remaining children were moved under
PID 1 (init). The only place where the grandchildren are still reachable
Quoted 2 lines
this way is in SIGCHLD handler while the dead child process still exists in zombie state.
Signal communicatoin is asynchronous processing, setup SIGCHILD handler can not 100% garantee the libarary process response in time as well.
Though the test_pid being moved under PID 1(init), kill(-test_pid, SIGKILL) still works well for killing them. That beacuse the dead child process still exists until kernel recliam its all parent.
Signature · 2 lines
Regards, Li Wang
Signature · 2 lines
-- Mailing list info: https://lists.linux.it/listinfo/ltp
Quoted 42 lines · Li Wang
Quoted 28 lines · Martin Doucha
Quoted 27 lines · Li Wang
On Fri, Feb 11, 2022 at 12:18 AM Martin Doucha <[email protected] <mailto:[email protected]>> wrote:
@@ -1560,6 +1568,7 @@ void tst_run_tcases(int argc, char *argv[], struct tst_test *self) SAFE_SIGNAL(SIGALRM, alarm_handler); SAFE_SIGNAL(SIGUSR1, heartbeat_handler); + SAFE_SIGNAL(SIGCHLD, sigchild_handler);Do we really need setup this signal handler for SIGCHILD?
Since we have already called '
SAFE_WAITPID(test_pid, &status, 0)' in the library process (lib_pid) which rely on SIGCHILD as well. And even this handler will be called everytime when test exit normally.Maybe better just add a kill function to cleanup the remain descendants if main test process exit with abonormal status.
e.g.
lib/tst_test.c +0
-0@@ -1503,6 +1503,8 @@ static int fork_testrun(void)if (WIFEXITED (status) &&WEXITSTATUS (status))return WEXITSTATUS (status);Quoted 4 lines
lib/tst_test.c static int fork_testrun(void) · line 1506
Quoted 1 line
lib/tst_test.c +1
-0+kill (-test_pid,SIGKILL );This will not work because at this point, the child process was already destroyed by
waitpid()and all its remaining children were moved underPID 1 (init). The only place where the grandchildren are still reachable
Quoted 2 lines
this way is in SIGCHLD handler while the dead child process still exists in zombie state.
Quoted 9 lines
Signal communicatoin is asynchronous processing, setup SIGCHILD handler can not 100% garantee the libarary process response in time as well.
Though the
test_pidbeing moved under PID 1(init),kill(-test_pid, SIGKILL)still works well for killing them. That beacuse the dead child process still
exists until kernel recliam its all parent.
I give 5 seconds sleep before sending SIGKILL in lib-process and modified the test_children_cleanup.c to print ppid each 1sec to verify this:
# ./test_children_cleanup tst_test.c:1452: TINFO: Timeout per run is 0h 00m 10s test_children_cleanup.c:20: TINFO: Main process 173236 starting test_children_cleanup.c:39: TINFO: Forked child 173238 test_children_cleanup.c:33: TINFO: ppid is 173236 test_children_cleanup.c:33: TINFO: ppid is 1 test_children_cleanup.c:33: TINFO: ppid is 1 test_children_cleanup.c:33: TINFO: ppid is 1 test_children_cleanup.c:33: TINFO: ppid is 1 tst_test.c:1502: TINFO: If you are running on slow machine, try exporting LTP_TIMEOUT_MUL > 1 tst_test.c:1504: TBROK: Test killed! (timeout?)
Summary:
passed 0 failed 0 broken 1 skipped 0
warnings 0
=======
lib/newlib_tests/test_children_cleanup.c +5 -1
@@ -28,7 +28,11 @@ static void run(void)/* Start child that will outlive the main test process */ if (!child_pid) { -sleep (30 ); +int i; +for (i =0 ; i <30 ; i++) { +tst_res (TINFO ,"ppid is %d" ,getppid ()); +sleep (1 ); + }return ; }
lib/tst_test.c +3 -0
@@ -1503,6 +1503,9 @@ static int fork_testrun(void)if (WIFEXITED (status) &&WEXITSTATUS (status))return WEXITSTATUS (status); +sleep (5 ); +kill (-test_pid,SIGKILL ); +if (WIFSIGNALED (status) &&WTERMSIG (status) ==SIGKILL ) {tst_res (TINFO ,"If you are running on slow machine, " "try exporting LTP_TIMEOUT_MUL > 1" );
Signature · 2 lines
Regards, Li Wang
Signature · 2 lines
-- Mailing list info: https://lists.linux.it/listinfo/ltp
Quoted 17 lines · Li Wang
I give 5 seconds sleep before sending SIGKILL in lib-process and modified the
test_children_cleanup.cto print ppid each 1sec to verify this:# ./test_children_cleanup tst_test.c:1452: TINFO: Timeout per run is 0h 00m 10s test_children_cleanup.c:20: TINFO: Main process 173236 starting test_children_cleanup.c:39: TINFO: Forked child 173238 test_children_cleanup.c:33: TINFO: ppid is 173236 test_children_cleanup.c:33: TINFO: ppid is 1 test_children_cleanup.c:33: TINFO: ppid is 1 test_children_cleanup.c:33: TINFO: ppid is 1 test_children_cleanup.c:33: TINFO: ppid is 1 tst_test.c:1502: TINFO: If you are running on slow machine, try exporting LTP_TIMEOUT_MUL > 1 tst_test.c:1504: TBROK: Test killed! (timeout?)
Quoted 7 lines
Summary:
passed 0
failed 0
broken 1
skipped 0
warnings 0
=======
Hmm, that's weird, when I tried that approach yesterday, it kept leaving the child behind. Now it seems to be working. I guess I messed up somehow and the test program didn't get relinked against new libltp.a...
OK, sending v2 soon since I also need to fix a bug in the libtest shell script.
Signature · 10 lines
Martin Doucha [email protected] QA Engineer for Software Maintenance SUSE LINUX, s.r.o. CORSO IIa Krizikova 148/34 186 00 Prague 8 Czech Republic -- Mailing list info: https://lists.linux.it/listinfo/ltp
Hi!
Quoted 3 lines
Hmm, that's weird, when I tried that approach yesterday, it kept leaving the child behind. Now it seems to be working. I guess I messed up somehow and the test program didn't get relinked against new libltp.a...
Just FYI the process group lifetime ends with the last process exitting it's group. Otherwise there could be all kind of races in the case when you send the SIGTERM signal to the whole group, e.g. if the leader got the signal first the rest of the processes may not be terminated if the group lifetime ended with the leader...
Signature · 5 lines
Cyril Hrubis [email protected] -- Mailing list info: https://lists.linux.it/listinfo/ltp