Skip to contentSkip to conversation

Terminate leftover subprocesses when main test process crashes

PATCH Superseded Martin Doucha 4 participants 9 messages Feb 10, 2022 – Feb 11, 2022 Library code

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 Terminate leftover subprocesses when main test process crashes

When the main test process crashes or gets killed e.g. by OOM killer, the watchdog process currently does not clean up any remaining child processes. Fix this by sending SIGKILL to the test process group when the watchdog process gets notified that the main test process has exited for any reason.

Fixes #892

---
 lib/newlib_tests/runtest.sh               |  3 +-
 lib/newlib_tests/test_children_cleanup.c  | 43 +++++++++++++++++++++++
 lib/newlib_tests/test_children_cleanup.sh | 17 +++++++++
 lib/tst_test.c                            |  9 +++++
 4 files changed, 71 insertions(+), 1 deletion(-)
 create mode 100644 lib/newlib_tests/test_children_cleanup.c
 create mode 100755 lib/newlib_tests/test_children_cleanup.sh
lib/newlib_tests/runtest.sh +2 -1
index 92fd3860e..327460e7b 100755
@@ -4,7 +4,8 @@
 LTP_C_API_TESTS="${LTP_C_API_TESTS:-test05 test07 test09 test12 test15 test18
 tst_needs_cmds01 tst_needs_cmds02 tst_needs_cmds03 tst_needs_cmds06
 tst_needs_cmds07 tst_bool_expr test_exec test_timer tst_res_hexd tst_strstatus
-tst_fuzzy_sync03 test_zero_hugepage.sh test_kconfig.sh}"
+tst_fuzzy_sync03 test_zero_hugepage.sh test_kconfig.sh
+test_children_cleanup.sh}"
 
 LTP_SHELL_API_TESTS="${LTP_SHELL_API_TESTS:-shell/tst_check_driver.sh
 shell/tst_check_kconfig0[1-5].sh shell/net/*.sh}"
lib/newlib_tests/test_children_cleanup.c new file +43 -0

43 lines not on this page · raw message

lib/newlib_tests/test_children_cleanup.sh new file +17 -0
new file mode 100755
index 000000000..d99b9b3a2
@@ -0,0 +1,17 @@
+#!/bin/sh
+# SPDX-License-Identifier: GPL-2.0-or-later
+# Copyright (c) 2022 SUSE LLC <[email protected]>
+
+TMPFILE="/tmp/ltp_children_cleanup_$$.log"
+CHILD_PID=`./test_children_cleanup 2>&1 | sed -n 's/^.*Forked child \([0-9]*\)$/\1/p'`
+
+if [ "x$CHILD_PID" = "x" ]; then
+	echo "TFAIL: Child process was not created"
+	exit 1
+elif ! kill -s 0 $CHILD_PID &>/dev/null; then
+	echo "TPASS: Child process was cleaned up"
+	exit 0
+else
+	echo "TFAIL: Child process was left behind"
+	exit 1
+fi
lib/tst_test.c +9 -0
index 191a9deab..09f65ac71 100644
@@ -1399,6 +1399,13 @@ static void sigint_handler(int sig LTP_ATTRIBUTE_UNUSED)
 	}
 }
 
+static void sigchild_handler(int sig LTP_ATTRIBUTE_UNUSED)
+{
+	/* If the test process is dead, send SIGKILL to its children */
+	if (kill(test_pid, 0))
+		kill(-test_pid, SIGKILL);
+}
+
 unsigned int tst_timeout_remaining(void)
 {
 	static struct timespec now;
@@ -1481,6 +1488,7 @@ static int fork_testrun(void)
 		tst_disable_oom_protection(0);
 		SAFE_SIGNAL(SIGALRM, SIG_DFL);
 		SAFE_SIGNAL(SIGUSR1, SIG_DFL);
+		SAFE_SIGNAL(SIGCHLD, SIG_DFL);
 		SAFE_SIGNAL(SIGINT, SIG_DFL);
 		SAFE_SETPGID(0, 0);
 		testrun();
@@ -1560,6 +1568,7 @@ void tst_run_tcases(int argc, char *argv[], struct tst_test *self)
 
 	SAFE_SIGNAL(SIGALRM, alarm_handler);
 	SAFE_SIGNAL(SIGUSR1, heartbeat_handler);
+	SAFE_SIGNAL(SIGCHLD, sigchild_handler);
 
 	if (tst_test->test_variants)
 		test_variants = tst_test->test_variants;
Signature · 5 lines
2.34.1


-- 
Mailing list info: https://lists.linux.it/listinfo/ltp
#1 lore raw
Quoted 34 lines · Martin Doucha

When the main test process crashes or gets killed e.g. by OOM killer, the watchdog process currently does not clean up any remaining child processes. Fix this by sending SIGKILL to the test process group when the watchdog process gets notified that the main test process has exited for any reason.

lib/tst_test.c +7 -0
@@ -1399,6 +1399,13 @@ static void sigint_handler(int sig
 LTP_ATTRIBUTE_UNUSED)
        }
 }
 
+static void sigchild_handler(int sig LTP_ATTRIBUTE_UNUSED)
+{
+       /* If the test process is dead, send SIGKILL to its children */
+       if (kill(test_pid, 0))
+               kill(-test_pid, SIGKILL);
+}
+
 unsigned int tst_timeout_remaining(void)
 {
        static struct timespec now;
lib/tst_test.c +1 -0
@@ -1481,6 +1488,7 @@ static int fork_testrun(void)
                tst_disable_oom_protection(0);
                SAFE_SIGNAL(SIGALRM, SIG_DFL);
                SAFE_SIGNAL(SIGUSR1, SIG_DFL);
+               SAFE_SIGNAL(SIGCHLD, SIG_DFL);
                SAFE_SIGNAL(SIGINT, SIG_DFL);
                SAFE_SETPGID(0, 0);
                testrun();

lib/tst_test.c void tst_run_tcases(int argc, char *argv[], struct · line 1573

Quoted 6 lines
lib/tst_test.c +1 -0
@@ -1560,6 +1568,7 @@ void tst_run_tcases(int argc, char *argv[], struct
 tst_test *self)
 
        SAFE_SIGNAL(SIGALRM, alarm_handler);
        SAFE_SIGNAL(SIGUSR1, heartbeat_handler);
+       SAFE_SIGNAL(SIGCHLD, sigchild_handler);
 

Do we really need setup this signal handler for SIGCHILD?

Since we have already called 'SAFE_WAITPID(test_pid, &status, 0)' in the library process (lib_pid) which rely on SIGCHILD as well. And even this handler will be called everytime when test exit normally.

Maybe better just add a kill function to cleanup the remain descendants if main test process exit with abonormal status.

e.g.

lib/tst_test.c +2 -0
@@ -1503,6 +1503,8 @@ static int fork_testrun(void)
        if (WIFEXITED(status) && WEXITSTATUS(status))
                return WEXITSTATUS(status);
 
+       kill(-test_pid, SIGKILL);
+
        if (WIFSIGNALED(status) && WTERMSIG(status) == SIGKILL) {
                tst_res(TINFO, "If you are running on slow machine, "
                               "try exporting LTP_TIMEOUT_MUL > 1");
Signature · 2 lines
Regards,
Li Wang
Signature · 2 lines
-- 
Mailing list info: https://lists.linux.it/listinfo/ltp
#2 lore raw
Quoted 39 lines · Li Wang
Quoted 35 lines · Martin Doucha

When the main test process crashes or gets killed e.g. by OOM killer, the watchdog process currently does not clean up any remaining child processes. Fix this by sending SIGKILL to the test process group when the watchdog process gets notified that the main test process has exited for any reason.

lib/tst_test.c +8 -0
@@ -1399,6 +1399,13 @@ static void sigint_handler(int sig LTP_ATTRIBUTE_UNUSED)
        }
 }
 
+static void sigchild_handler(int sig LTP_ATTRIBUTE_UNUSED)
+{
+       /* If the test process is dead, send SIGKILL to its children */
+       if (kill(test_pid, 0))
+               kill(-test_pid, SIGKILL);
+}
+
 unsigned int tst_timeout_remaining(void)
 {
        static struct timespec now;
@@ -1481,6 +1488,7 @@ static int fork_testrun(void)
                tst_disable_oom_protection(0);
                SAFE_SIGNAL(SIGALRM, SIG_DFL);
                SAFE_SIGNAL(SIGUSR1, SIG_DFL);
+               SAFE_SIGNAL(SIGCHLD, SIG_DFL);
                SAFE_SIGNAL(SIGINT, SIG_DFL);
                SAFE_SETPGID(0, 0);
                testrun();
@@ -1560,6 +1568,7 @@ void tst_run_tcases(int argc, char *argv[], struct tst_test *self)
 
Quoted 6 lines

lib/tst_test.c void tst_run_tcases(int argc, char *argv[], struct tst_test *self) · line 1571

Quoted 3 lines
lib/tst_test.c +1 -0
        SAFE_SIGNAL(SIGALRM, alarm_handler);
        SAFE_SIGNAL(SIGUSR1, heartbeat_handler);
+       SAFE_SIGNAL(SIGCHLD, sigchild_handler);

Do we really need setup this signal handler for SIGCHILD?

I had same question.

Quoted 13 lines

Since we have already called 'SAFE_WAITPID(test_pid, &status, 0)' in the library process (lib_pid) which rely on SIGCHILD as well. And even this handler will be called everytime when test exit normally.

Maybe better just add a kill function to cleanup the remain descendants if main test process exit with abonormal status.

e.g.

lib/tst_test.c static int fork_testrun(void) · line 1506

Quoted 4 lines
lib/tst_test.c +1 -0
@@ -1503,6 +1503,8 @@ static int fork_testrun(void)
        if (WIFEXITED(status) && WEXITSTATUS(status))
                return WEXITSTATUS(status);
 
+       kill(-test_pid, SIGKILL);

Could we skip the call if forks_child == 0 ?

Quoted 11 lines
lib/tst_test.c +1 -1
+
        if (WIFSIGNALED(status) && WTERMSIG(status) == SIGKILL) {
                tst_res(TINFO, "If you are running on slow machine, "
                               "try exporting LTP_TIMEOUT_MUL > 1");
 
--

Regards,
Li Wang

--
Mailing list info: https://lists.linux.it/listinfo/ltp

Signature · 1 line
Mailing list info: https://lists.linux.it/listinfo/ltp
#3 lore raw
Quoted 67 lines · Jan Stancek
Quoted 13 lines · Li Wang
Quoted 9 lines · Martin Doucha

When the main test process crashes or gets killed e.g. by OOM killer, the watchdog process currently does not clean up any remaining child processes. Fix this by sending SIGKILL to the test process group when the watchdog process gets notified that the main test process has exited for any reason.

Quoted 3 lines

lib/tst_test.c static void sigint_handler(int sig

Quoted 3 lines
lib/tst_test.c +0 -0
@@ -1399,6 +1399,13 @@ static void sigint_handler(int sig

LTP_ATTRIBUTE_UNUSED)

Quoted 14 lines
Quoted 14 lines
lib/tst_test.c +7 -0
        }
 }
 
+static void sigchild_handler(int sig LTP_ATTRIBUTE_UNUSED)
+{
+       /* If the test process is dead, send SIGKILL to its children */
+       if (kill(test_pid, 0))
+               kill(-test_pid, SIGKILL);
+}
+
 unsigned int tst_timeout_remaining(void)
 {
        static struct timespec now;
@@ -1481,6 +1488,7 @@ static int fork_testrun(void)
Quoted 8 lines

lib/tst_test.c void tst_run_tcases(int argc, char *argv[],

Quoted 8 lines
lib/tst_test.c +1 -0
                tst_disable_oom_protection(0);
                SAFE_SIGNAL(SIGALRM, SIG_DFL);
                SAFE_SIGNAL(SIGUSR1, SIG_DFL);
+               SAFE_SIGNAL(SIGCHLD, SIG_DFL);
                SAFE_SIGNAL(SIGINT, SIG_DFL);
                SAFE_SETPGID(0, 0);
                testrun();
@@ -1560,6 +1568,7 @@ void tst_run_tcases(int argc, char *argv[],

struct tst_test *self)

Quoted 7 lines

lib/tst_test.c void tst_run_tcases(int argc, char *argv[], · line 1570

Quoted 4 lines
lib/tst_test.c +1 -0
        SAFE_SIGNAL(SIGALRM, alarm_handler);
        SAFE_SIGNAL(SIGUSR1, heartbeat_handler);
+       SAFE_SIGNAL(SIGCHLD, sigchild_handler);

Do we really need setup this signal handler for SIGCHILD?

I had same question.

Quoted 16 lines

Since we have already called 'SAFE_WAITPID(test_pid, &status, 0)' in the library process (lib_pid) which rely on SIGCHILD as well. And even this handler will be called everytime when test exit normally.

Maybe better just add a kill function to cleanup the remain descendants if main test process exit with abonormal status.

e.g.

lib/tst_test.c +0 -0
@@ -1503,6 +1503,8 @@ static int fork_testrun(void)
        if (WIFEXITED(status) && WEXITSTATUS(status))
                return WEXITSTATUS(status);
 
Quoted 4 lines

lib/tst_test.c static int fork_testrun(void) · line 1506

Quoted 1 line
lib/tst_test.c +1 -0
+       kill(-test_pid, SIGKILL);

Could we skip the call if forks_child == 0 ?

+1 Obviously yes!

Signature · 2 lines
Regards,
Li Wang
Signature · 2 lines
-- 
Mailing list info: https://lists.linux.it/listinfo/ltp
#4 lore raw
In reply to Li Wang
Quoted 26 lines · Li Wang

On Fri, Feb 11, 2022 at 12:18 AM Martin Doucha <[email protected] <mailto:[email protected]>> wrote:

    @@ -1560,6 +1568,7 @@ void tst_run_tcases(int argc, char *argv[],
    struct tst_test *self)

            SAFE_SIGNAL(SIGALRM, alarm_handler);
            SAFE_SIGNAL(SIGUSR1, heartbeat_handler);
    +       SAFE_SIGNAL(SIGCHLD, sigchild_handler);

Do we really need setup this signal handler for SIGCHILD?

Since we have already called 'SAFE_WAITPID(test_pid, &status, 0)' in the library process (lib_pid) which rely on SIGCHILD as well. And even this handler will be called everytime when test exit normally.

Maybe better just add a kill function to cleanup the remain descendants if main test process exit with abonormal status.

e.g.

lib/tst_test.c +0 -0
@@ -1503,6 +1503,8 @@ static int fork_testrun(void)
         if (WIFEXITED(status) && WEXITSTATUS(status))
                 return WEXITSTATUS(status);
Quoted 2 lines


+       kill(-test_pid, SIGKILL);

This will not work because at this point, the child process was already destroyed by waitpid() and all its remaining children were moved under PID 1 (init). The only place where the grandchildren are still reachable this way is in SIGCHLD handler while the dead child process still exists in zombie state.

Signature · 10 lines
Martin Doucha   [email protected]
QA Engineer for Software Maintenance
SUSE LINUX, s.r.o.
CORSO IIa
Krizikova 148/34
186 00 Prague 8
Czech Republic

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp
#5 lore raw
Quoted 28 lines · Martin Doucha
Quoted 27 lines · Li Wang

On Fri, Feb 11, 2022 at 12:18 AM Martin Doucha <[email protected] <mailto:[email protected]>> wrote:

    @@ -1560,6 +1568,7 @@ void tst_run_tcases(int argc, char *argv[],
    struct tst_test *self)

            SAFE_SIGNAL(SIGALRM, alarm_handler);
            SAFE_SIGNAL(SIGUSR1, heartbeat_handler);
    +       SAFE_SIGNAL(SIGCHLD, sigchild_handler);

Do we really need setup this signal handler for SIGCHILD?

Since we have already called 'SAFE_WAITPID(test_pid, &status, 0)' in the library process (lib_pid) which rely on SIGCHILD as well. And even this handler will be called everytime when test exit normally.

Maybe better just add a kill function to cleanup the remain descendants if main test process exit with abonormal status.

e.g.

lib/tst_test.c +0 -0
@@ -1503,6 +1503,8 @@ static int fork_testrun(void)
        if (WIFEXITED(status) && WEXITSTATUS(status))
                return WEXITSTATUS(status);
 
Quoted 4 lines

lib/tst_test.c static int fork_testrun(void) · line 1506

Quoted 1 line
lib/tst_test.c +1 -0
+       kill(-test_pid, SIGKILL);

This will not work because at this point, the child process was already destroyed by waitpid() and all its remaining children were moved under

PID 1 (init). The only place where the grandchildren are still reachable

Quoted 2 lines

this way is in SIGCHLD handler while the dead child process still exists in zombie state.

Signal communicatoin is asynchronous processing, setup SIGCHILD handler can not 100% garantee the libarary process response in time as well.

Though the test_pid being moved under PID 1(init), kill(-test_pid, SIGKILL) still works well for killing them. That beacuse the dead child process still exists until kernel recliam its all parent.

Signature · 2 lines
Regards,
Li Wang
Signature · 2 lines
-- 
Mailing list info: https://lists.linux.it/listinfo/ltp
#6 lore raw
Quoted 42 lines · Li Wang
Quoted 28 lines · Martin Doucha
Quoted 27 lines · Li Wang

On Fri, Feb 11, 2022 at 12:18 AM Martin Doucha <[email protected] <mailto:[email protected]>> wrote:

    @@ -1560,6 +1568,7 @@ void tst_run_tcases(int argc, char *argv[],
    struct tst_test *self)

            SAFE_SIGNAL(SIGALRM, alarm_handler);
            SAFE_SIGNAL(SIGUSR1, heartbeat_handler);
    +       SAFE_SIGNAL(SIGCHLD, sigchild_handler);

Do we really need setup this signal handler for SIGCHILD?

Since we have already called 'SAFE_WAITPID(test_pid, &status, 0)' in the library process (lib_pid) which rely on SIGCHILD as well. And even this handler will be called everytime when test exit normally.

Maybe better just add a kill function to cleanup the remain descendants if main test process exit with abonormal status.

e.g.

lib/tst_test.c +0 -0
@@ -1503,6 +1503,8 @@ static int fork_testrun(void)
        if (WIFEXITED(status) && WEXITSTATUS(status))
                return WEXITSTATUS(status);
 
Quoted 4 lines

lib/tst_test.c static int fork_testrun(void) · line 1506

Quoted 1 line
lib/tst_test.c +1 -0
+       kill(-test_pid, SIGKILL);

This will not work because at this point, the child process was already destroyed by waitpid() and all its remaining children were moved under

PID 1 (init). The only place where the grandchildren are still reachable

Quoted 2 lines

this way is in SIGCHLD handler while the dead child process still exists in zombie state.

Quoted 9 lines

Signal communicatoin is asynchronous processing, setup SIGCHILD handler can not 100% garantee the libarary process response in time as well.

Though the test_pid being moved under PID 1(init), kill(-test_pid, SIGKILL) still works well for killing them. That beacuse the dead child process still
exists until kernel recliam its all parent.

I give 5 seconds sleep before sending SIGKILL in lib-process and modified the test_children_cleanup.c to print ppid each 1sec to verify this:

# ./test_children_cleanup
tst_test.c:1452: TINFO: Timeout per run is 0h 00m 10s
test_children_cleanup.c:20: TINFO: Main process 173236 starting
test_children_cleanup.c:39: TINFO: Forked child 173238
test_children_cleanup.c:33: TINFO: ppid is 173236
test_children_cleanup.c:33: TINFO: ppid is 1
test_children_cleanup.c:33: TINFO: ppid is 1
test_children_cleanup.c:33: TINFO: ppid is 1
test_children_cleanup.c:33: TINFO: ppid is 1
tst_test.c:1502: TINFO: If you are running on slow machine, try exporting
LTP_TIMEOUT_MUL > 1
tst_test.c:1504: TBROK: Test killed! (timeout?)

Summary:

passed   0
failed   0
broken   1
skipped  0

warnings 0
=======

lib/newlib_tests/test_children_cleanup.c +5 -1
@@ -28,7 +28,11 @@ static void run(void)
 
        /* Start child that will outlive the main test process */
        if (!child_pid) {
-               sleep(30);
+               int i;
+               for (i = 0; i < 30; i++) {
+                       tst_res(TINFO, "ppid is %d", getppid());
+                       sleep(1);
+               }
                return;
        }
 
lib/tst_test.c +3 -0
index 84ce0a5d3..6f2d93611 100644
@@ -1503,6 +1503,9 @@ static int fork_testrun(void)
        if (WIFEXITED(status) && WEXITSTATUS(status))
                return WEXITSTATUS(status);
 
+       sleep(5);
+       kill(-test_pid, SIGKILL);
+
        if (WIFSIGNALED(status) && WTERMSIG(status) == SIGKILL) {
                tst_res(TINFO, "If you are running on slow machine, "
                               "try exporting LTP_TIMEOUT_MUL > 1");
Signature · 2 lines
Regards,
Li Wang
Signature · 2 lines
-- 
Mailing list info: https://lists.linux.it/listinfo/ltp
#7 lore raw
Quoted 17 lines · Li Wang

I give 5 seconds sleep before sending SIGKILL in lib-process and modified the test_children_cleanup.c to print ppid each 1sec to verify this:

# ./test_children_cleanup
tst_test.c:1452: TINFO: Timeout per run is 0h 00m 10s
test_children_cleanup.c:20: TINFO: Main process 173236 starting
test_children_cleanup.c:39: TINFO: Forked child 173238
test_children_cleanup.c:33: TINFO: ppid is 173236
test_children_cleanup.c:33: TINFO: ppid is 1
test_children_cleanup.c:33: TINFO: ppid is 1
test_children_cleanup.c:33: TINFO: ppid is 1
test_children_cleanup.c:33: TINFO: ppid is 1
tst_test.c:1502: TINFO: If you are running on slow machine, try
exporting LTP_TIMEOUT_MUL > 1
tst_test.c:1504: TBROK: Test killed! (timeout?)
Quoted 7 lines

Summary:
passed   0
failed   0
broken   1
skipped  0
warnings 0
=======

Hmm, that's weird, when I tried that approach yesterday, it kept leaving the child behind. Now it seems to be working. I guess I messed up somehow and the test program didn't get relinked against new libltp.a...

OK, sending v2 soon since I also need to fix a bug in the libtest shell script.

Signature · 10 lines
Martin Doucha   [email protected]
QA Engineer for Software Maintenance
SUSE LINUX, s.r.o.
CORSO IIa
Krizikova 148/34
186 00 Prague 8
Czech Republic

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp
#8 lore raw

Hi!

Quoted 3 lines

Hmm, that's weird, when I tried that approach yesterday, it kept leaving the child behind. Now it seems to be working. I guess I messed up somehow and the test program didn't get relinked against new libltp.a...

Just FYI the process group lifetime ends with the last process exitting it's group. Otherwise there could be all kind of races in the case when you send the SIGTERM signal to the whole group, e.g. if the leader got the signal first the rest of the processes may not be terminated if the group lifetime ended with the leader...

Signature · 5 lines
Cyril Hrubis
[email protected]

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp