Skip to contentSkip to conversation

test_children_cleanup.sh: Fix race condition

PATCH Superseded Martin Doucha 3 participants 6 messages Feb 14, 2022 – Feb 15, 2022

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 test_children_cleanup.sh: Fix race condition

Processes can stay alive for a short while even after receiving SIGKILL. Give the child in subprocess cleanup libtest up to 5 seconds to fully exit before reporting that it was left behind.

---
 lib/newlib_tests/test_children_cleanup.sh | 21 +++++++++++++++------
 1 file changed, 15 insertions(+), 6 deletions(-)
lib/newlib_tests/test_children_cleanup.sh +15 -6
index 4b4e8b2f0..44505aa51 100755
@@ -10,10 +10,19 @@ rm "$TMPFILE"
 if [ "x$CHILD_PID" = "x" ]; then
 	echo "TFAIL: Child process was not created"
 	exit 1
-elif ! kill -s 0 $CHILD_PID &>/dev/null; then
-	echo "TPASS: Child process was cleaned up"
-	exit 0
-else
-	echo "TFAIL: Child process was left behind"
-	exit 1
 fi
+
+# The child process can stay alive for a short while even after receiving
+# SIGKILL, especially if the system is under heavy load. Wait up to 5 seconds
+# for it to fully exit.
+for i in `seq 6`; do
+	if ! [ -e "/proc/$CHILD_PID" ]; then
+		echo "TPASS: Child process was cleaned up"
+		exit 0
+	fi
+
+	sleep 1
+done
+
+echo "TFAIL: Child process was left behind"
+exit 1
Signature · 5 lines
2.34.1


-- 
Mailing list info: https://lists.linux.it/listinfo/ltp
#1 lore raw
Quoted 29 lines · Martin Doucha

Processes can stay alive for a short while even after receiving SIGKILL. Give the child in subprocess cleanup libtest up to 5 seconds to fully exit before reporting that it was left behind.

---
 lib/newlib_tests/test_children_cleanup.sh | 21 +++++++++++++++------
 1 file changed, 15 insertions(+), 6 deletions(-)
lib/newlib_tests/test_children_cleanup.sh +3 -6
index 4b4e8b2f0..44505aa51 100755
@@ -10,10 +10,19 @@ rm "$TMPFILE"
 if [ "x$CHILD_PID" = "x" ]; then
        echo "TFAIL: Child process was not created"
        exit 1
-elif ! kill -s 0 $CHILD_PID &>/dev/null; then
-       echo "TPASS: Child process was cleaned up"
-       exit 0
-else
-       echo "TFAIL: Child process was left behind"
-       exit 1
 fi
+
+# The child process can stay alive for a short while even after receiving
+# SIGKILL, especially if the system is under heavy load. Wait up to 5
Quoted 3 lines

seconds
+# for it to fully exit.

It doesn't work for all platforms and we can not guarantee how long it will cost before PID 1 reaps zombie process.

Also, I just get to know that Docker does not run processes under a special init process that properly reaps child processes, so that it is possible for the container to end up with zombie processes that cause all sorts of trouble.

I even try adding kill -SIGCHLD 1 but does not work as expected.

See CI jobs:
https://github.com/wangli5665/ltp/runs/5194270998?check_suite_focus=true
<https://mail.google.com/mail/u/1/%E2%80%8Bhttps://github.com/wangli5665/ltp/runs/5194270998?check_suite_focus=true>

Therefore, I suggest giving a chance to my refined patch V2 :).

lib/newlib_tests/test_children_cleanup.sh +6 -0
@@ -10,10 +10,16 @@ rm "$TMPFILE"
 if [ "x$CHILD_PID" = "x" ]; then
        echo "TFAIL: Child process was not created"
        exit 1
+elif grep -q "Z (zombie)" /proc/$CHILD_PID/status; then
+       echo "TPASS: Child process is in zombie state"
+       exit 0
 elif ! kill -s 0 $CHILD_PID &>/dev/null; then
        echo "TPASS: Child process was cleaned up"
        exit 0
 else
        echo "TFAIL: Child process was left behind"
+       echo "cat /proc/$CHILD_PID/status"
+       echo "---------------------------"
+       cat /proc/$CHILD_PID/status
        exit 1
 fi
Signature · 2 lines
Regards,
Li Wang
Signature · 2 lines
-- 
Mailing list info: https://lists.linux.it/listinfo/ltp
#2 lore raw

Hi all,

...

Quoted 25 lines

It doesn't work for all platforms and we can not guarantee how long it will cost before PID 1 reaps zombie process.

Also, I just get to know that Docker does not run processes under a special init process that properly reaps child processes, so that it is possible for the container to end up with zombie processes that cause all sorts of trouble.

I even try adding kill -SIGCHLD 1 but does not work as expected.

See CI jobs:
https://github.com/wangli5665/ltp/runs/5194270998?check_suite_focus=true
<https://mail.google.com/mail/u/1/%E2%80%8Bhttps://github.com/wangli5665/ltp/runs/5194270998?check_suite_focus=true>

Therefore, I suggest giving a chance to my refined patch V2 :).

lib/newlib_tests/test_children_cleanup.sh +3 -0
@@ -10,10 +10,16 @@ rm "$TMPFILE"
 if [ "x$CHILD_PID" = "x" ]; then
        echo "TFAIL: Child process was not created"
        exit 1
+elif grep -q "Z (zombie)" /proc/$CHILD_PID/status; then
+       echo "TPASS: Child process is in zombie state"
+       exit 0

lib/newlib_tests/test_children_cleanup.sh rm "$TMPFILE" · line 25

Quoted 10 lines
lib/newlib_tests/test_children_cleanup.sh +3 -0
 elif ! kill -s 0 $CHILD_PID &>/dev/null; then
        echo "TPASS: Child process was cleaned up"
        exit 0
 else
        echo "TFAIL: Child process was left behind"
+       echo "cat /proc/$CHILD_PID/status"
+       echo "---------------------------"
+       cat /proc/$CHILD_PID/status
        exit 1
 fi

Li's approach LGTM.

Kind regards,
Petr

Signature · 1 line
Mailing list info: https://lists.linux.it/listinfo/ltp
#3 lore raw
In reply to Li Wang
Quoted 8 lines · Li Wang

It doesn't work for all platforms and we can not guarantee how long it will cost before PID 1 reaps zombie process.

Also, I just get to know that  Docker does not run processes under a special init process that properly reaps child processes, so that it is possible for the container to end up with zombie processes that cause all sorts of trouble.

Quoted 7 lines

I even try adding kill -SIGCHLD 1 but does not work as expected.

See CI jobs:
  https://github.com/wangli5665/ltp/runs/5194270998?check_suite_focus=true
<https://mail.google.com/mail/u/1/%E2%80%8Bhttps://github.com/wangli5665/ltp/runs/5194270998?check_suite_focus=true>

Therefore, I suggest giving a chance to my refined patch V2 :).

When I was testing the libtest yesterday on a moderately stressed machine, I actually saw the child process still in the R state during the first state check a couple of times. That's why I've added looping with delay.

On the other hand I did not see any zombies even after several hundred tries. But I can add a zombie check to my patch a well.

Signature · 10 lines
Martin Doucha   [email protected]
QA Engineer for Software Maintenance
SUSE LINUX, s.r.o.
CORSO IIa
Krizikova 148/34
186 00 Prague 8
Czech Republic

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp
#4 lore raw
Quoted 17 lines · Li Wang
Quoted 8 lines

It doesn't work for all platforms and we can not guarantee how long it will cost before PID 1 reaps zombie process.

Also, I just get to know that  Docker does not run processes under a special init process that properly reaps child processes, so that it is possible for the container to end up with zombie processes that cause all sorts of trouble.

Quoted 7 lines

I even try adding kill -SIGCHLD 1 but does not work as expected.

See CI jobs:
  https://github.com/wangli5665/ltp/runs/5194270998?check_suite_focus=true
<https://mail.google.com/mail/u/1/%E2%80%8Bhttps://github.com/wangli5665/ltp/runs/5194270998?check_suite_focus=true>

Therefore, I suggest giving a chance to my refined patch V2 :).

Quoted 7 lines

When I was testing the libtest yesterday on a moderately stressed machine, I actually saw the child process still in the R state during the first state check a couple of times. That's why I've added looping with delay.

On the other hand I did not see any zombies even after several hundred tries. But I can add a zombie check to my patch a well.

I'd be for it. As Li noticed, containers behave really differently (maybe it'd be faster to debug tests using podman).

Kind regards,
Petr

Signature · 1 line
Mailing list info: https://lists.linux.it/listinfo/ltp
#5 lore raw
Quoted 10 lines · Petr Vorel
Quoted 2 lines · Li Wang
Quoted 1 line

It doesn't work for all platforms and we can not guarantee how long it

will

Quoted 6 lines
Quoted 6 lines

cost before PID 1 reaps zombie process.

Also, I just get to know that Docker does not run processes under a special init process that properly reaps child processes, so that it is possible for the container to end up with zombie processes that cause all sorts of trouble.

Quoted 14 lines
Quoted 4 lines
Quoted 4 lines

I even try adding kill -SIGCHLD 1 but does not work as expected.

See CI jobs:

https://github.com/wangli5665/ltp/runs/5194270998?check_suite_focus=true

Quoted 1 line
Quoted 1 line

<

https://mail.google.com/mail/u/1/%E2%80%8Bhttps://github.com/wangli5665/ltp/runs/5194270998?check_suite_focus=true

Quoted 6 lines
Quoted 3 lines

Therefore, I suggest giving a chance to my refined patch V2 :).

When I was testing the libtest yesterday on a moderately stressed machine, I actually saw the child process still in the R state during

That might cause by signal asynchronous processing when the system is under pressure.

Quoted 9 lines
Quoted 6 lines

the first state check a couple of times. That's why I've added looping with delay.

On the other hand I did not see any zombies even after several hundred tries. But I can add a zombie check to my patch a well.

I'd be for it. As Li noticed, containers behave really differently (maybe it'd be faster to debug tests using podman).

Right, we have to consider many scenarios:

1. asynchronous processing signal so that child still in 'R'
2. the child correctly terminated on bare-metal system
3. allow zombied while running by docker (without a formal init(PID 1) process)

Especially scenario-3 confused me for quite a while.

@Martin, feel free to send the patch that includes the above. or I can help summarize these in the commit description.

Signature · 2 lines
Regards,
Li Wang
Signature · 2 lines
-- 
Mailing list info: https://lists.linux.it/listinfo/ltp