test_children_cleanup.sh: Fix race condition
PATCH 1/1 test_children_cleanup.sh: Fix race condition
Processes can stay alive for a short while even after receiving SIGKILL. Give the child in subprocess cleanup libtest up to 5 seconds to fully exit before reporting that it was left behind.
--- lib/newlib_tests/test_children_cleanup.sh | 21 +++++++++++++++------ 1 file changed, 15 insertions(+), 6 deletions(-)
lib/newlib_tests/test_children_cleanup.sh +15 -6
@@ -10,10 +10,19 @@ rm "$TMPFILE" if [ "x$CHILD_PID" = "x" ]; then echo "TFAIL: Child process was not created" exit 1 -elif ! kill -s 0 $CHILD_PID &>/dev/null; then - echo "TPASS: Child process was cleaned up" - exit 0 -else - echo "TFAIL: Child process was left behind" - exit 1 fi + +# The child process can stay alive for a short while even after receiving +# SIGKILL, especially if the system is under heavy load. Wait up to 5 seconds +# for it to fully exit. +for i in `seq 6`; do + if ! [ -e "/proc/$CHILD_PID" ]; then + echo "TPASS: Child process was cleaned up" + exit 0 + fi + + sleep 1 +done + +echo "TFAIL: Child process was left behind" +exit 1
Signature · 5 lines
2.34.1 -- Mailing list info: https://lists.linux.it/listinfo/ltp
Quoted 29 lines · Martin Doucha
Processes can stay alive for a short while even after receiving SIGKILL. Give the child in subprocess cleanup libtest up to 5 seconds to fully exit before reporting that it was left behind.
Signed-off-by Martin Doucha <[email protected]>--- lib/newlib_tests/test_children_cleanup.sh | 21 +++++++++++++++------ 1 file changed, 15 insertions(+), 6 deletions(-)lib/newlib_tests/test_children_cleanup.sh +3
-6@@ -10,10 +10,19 @@ rm "$TMPFILE" if [ "x$CHILD_PID" = "x" ]; then echo "TFAIL: Child process was not created" exit 1 -elif ! kill -s 0 $CHILD_PID &>/dev/null; then - echo "TPASS: Child process was cleaned up" - exit 0 -else - echo "TFAIL: Child process was left behind" - exit 1 fi + +# The child process can stay alive for a short while even after receiving +# SIGKILL, especially if the system is under heavy load. Wait up to 5
Quoted 3 lines
seconds
+# for it to fully exit.
It doesn't work for all platforms and we can not guarantee how long it will cost before PID 1 reaps zombie process.
Also, I just get to know that Docker does not run processes under a special init process that properly reaps child processes, so that it is possible for the container to end up with zombie processes that cause all sorts of trouble.
I even try adding kill -SIGCHLD 1 but does not work as expected.
See CI jobs:
https://github.com/wangli5665/ltp/runs/5194270998?check_suite_focus=true
<https://mail.google.com/mail/u/1/%E2%80%8Bhttps://github.com/wangli5665/ltp/runs/5194270998?check_suite_focus=true>
Therefore, I suggest giving a chance to my refined patch V2 :).
lib/newlib_tests/test_children_cleanup.sh +6 -0
@@ -10,10 +10,16 @@ rm "$TMPFILE" if [ "x$CHILD_PID" = "x" ]; then echo "TFAIL: Child process was not created" exit 1 +elif grep -q "Z (zombie)" /proc/$CHILD_PID/status; then + echo "TPASS: Child process is in zombie state" + exit 0 elif ! kill -s 0 $CHILD_PID &>/dev/null; then echo "TPASS: Child process was cleaned up" exit 0 else echo "TFAIL: Child process was left behind" + echo "cat /proc/$CHILD_PID/status" + echo "---------------------------" + cat /proc/$CHILD_PID/status exit 1 fi
Signature · 2 lines
Regards, Li Wang
Signature · 2 lines
-- Mailing list info: https://lists.linux.it/listinfo/ltp
Hi all,
...
Quoted 25 lines
It doesn't work for all platforms and we can not guarantee how long it will cost before PID 1 reaps zombie process.
Also, I just get to know that Docker does not run processes under a special init process that properly reaps child processes, so that it is possible for the container to end up with zombie processes that cause all sorts of trouble.
I even try adding
kill -SIGCHLD 1but does not work as expected.See CI jobs:
https://github.com/wangli5665/ltp/runs/5194270998?check_suite_focus=true
<https://mail.google.com/mail/u/1/%E2%80%8Bhttps://github.com/wangli5665/ltp/runs/5194270998?check_suite_focus=true>Therefore, I suggest giving a chance to my refined patch V2 :).
lib/newlib_tests/test_children_cleanup.sh +3
-0@@ -10,10 +10,16 @@ rm "$TMPFILE" if [ "x$CHILD_PID" = "x" ]; then echo "TFAIL: Child process was not created" exit 1 +elif grep -q "Z (zombie)" /proc/$CHILD_PID/status; then + echo "TPASS: Child process is in zombie state" + exit 0
lib/newlib_tests/test_children_cleanup.sh rm "$TMPFILE" · line 25
Quoted 10 lines
lib/newlib_tests/test_children_cleanup.sh +3
-0elif ! kill -s 0 $CHILD_PID &>/dev/null; then echo "TPASS: Child process was cleaned up" exit 0 else echo "TFAIL: Child process was left behind" + echo "cat /proc/$CHILD_PID/status" + echo "---------------------------" + cat /proc/$CHILD_PID/status exit 1 fi
Li's approach LGTM.
Kind regards,
Petr
Signature · 1 line
Mailing list info: https://lists.linux.it/listinfo/ltp
Quoted 8 lines · Li Wang
It doesn't work for all platforms and we can not guarantee how long it will cost before PID 1 reaps zombie process.
Also, I just get to know that Docker does not run processes under a special init process that properly reaps child processes, so that it is possible for the container to end up with zombie processes that cause all sorts of trouble.
Quoted 7 lines
I even try adding
kill -SIGCHLD 1but does not work as expected.See CI jobs:
https://github.com/wangli5665/ltp/runs/5194270998?check_suite_focus=true
<https://mail.google.com/mail/u/1/%E2%80%8Bhttps://github.com/wangli5665/ltp/runs/5194270998?check_suite_focus=true>Therefore, I suggest giving a chance to my refined patch V2 :).
When I was testing the libtest yesterday on a moderately stressed machine, I actually saw the child process still in the R state during the first state check a couple of times. That's why I've added looping with delay.
On the other hand I did not see any zombies even after several hundred tries. But I can add a zombie check to my patch a well.
Signature · 10 lines
Martin Doucha [email protected] QA Engineer for Software Maintenance SUSE LINUX, s.r.o. CORSO IIa Krizikova 148/34 186 00 Prague 8 Czech Republic -- Mailing list info: https://lists.linux.it/listinfo/ltp
Quoted 17 lines · Li Wang
Quoted 8 lines
It doesn't work for all platforms and we can not guarantee how long it will cost before PID 1 reaps zombie process.
Also, I just get to know that Docker does not run processes under a special init process that properly reaps child processes, so that it is possible for the container to end up with zombie processes that cause all sorts of trouble.
Quoted 7 lines
I even try adding
kill -SIGCHLD 1but does not work as expected.See CI jobs:
https://github.com/wangli5665/ltp/runs/5194270998?check_suite_focus=true
<https://mail.google.com/mail/u/1/%E2%80%8Bhttps://github.com/wangli5665/ltp/runs/5194270998?check_suite_focus=true>Therefore, I suggest giving a chance to my refined patch V2 :).
Quoted 7 lines
When I was testing the libtest yesterday on a moderately stressed machine, I actually saw the child process still in the R state during the first state check a couple of times. That's why I've added looping with delay.
On the other hand I did not see any zombies even after several hundred tries. But I can add a zombie check to my patch a well.
I'd be for it. As Li noticed, containers behave really differently (maybe it'd be faster to debug tests using podman).
Kind regards,
Petr
Signature · 1 line
Mailing list info: https://lists.linux.it/listinfo/ltp
Quoted 10 lines · Petr Vorel
Quoted 2 lines · Li Wang
Quoted 1 line
It doesn't work for all platforms and we can not guarantee how long it
will
Quoted 6 lines
Quoted 6 lines
cost before PID 1 reaps zombie process.
Also, I just get to know that Docker does not run processes under a special init process that properly reaps child processes, so that it is possible for the container to end up with zombie processes that cause all sorts of trouble.
Quoted 14 lines
Quoted 4 lines
Quoted 4 lines
I even try adding
kill -SIGCHLD 1but does not work as expected.See CI jobs:
https://github.com/wangli5665/ltp/runs/5194270998?check_suite_focus=true
Quoted 1 line
Quoted 1 line
<
Quoted 6 lines
Quoted 3 lines
Therefore, I suggest giving a chance to my refined patch V2 :).
When I was testing the libtest yesterday on a moderately stressed machine, I actually saw the child process still in the R state during
That might cause by signal asynchronous processing when the system is under pressure.
Quoted 9 lines
Quoted 6 lines
the first state check a couple of times. That's why I've added looping with delay.
On the other hand I did not see any zombies even after several hundred tries. But I can add a zombie check to my patch a well.
I'd be for it. As Li noticed, containers behave really differently (maybe it'd be faster to debug tests using podman).
Right, we have to consider many scenarios:
1. asynchronous processing signal so that child still in 'R'
2. the child correctly terminated on bare-metal system
3. allow zombied while running by docker (without a formal init(PID 1) process)
Especially scenario-3 confused me for quite a while.
@Martin, feel free to send the patch that includes the above. or I can help summarize these in the commit description.
Signature · 2 lines
Regards, Li Wang
Signature · 2 lines
-- Mailing list info: https://lists.linux.it/listinfo/ltp