lib: ignore SIGINT in _tst_kill_test
PATCH v3 3/4 lib: ignore SIGINT in _tst_kill_test
We have to guarantee _tst_kill_test alive for a while to check if the target test eixst or not, so ignore SIGINT in _tst_kill_test is necessary, otherwise it will be stopped by the SIGINT sending by itself.
The timeout03.sh verify this mechanism proccess well in output:
timeout03 1 TBROK: Test timeouted, sending SIGINT! If you are running on slow machine, try exporting LTP_TIMEOUT_MUL > 1 timeout03 1 TBROK: test interrupted or timed out
timeout03 1 TPASS: test run cleanup after timeout
timeout03 1 TINFO: Test is still running, waiting 10s
timeout03 1 TINFO: Test is still running, waiting 9s
timeout03 1 TINFO: Test is still running, waiting 8s
timeout03 1 TINFO: Test is still running, waiting 7s
timeout03 1 TINFO: Test is still running, waiting 6s
timeout03 1 TINFO: Test is still running, waiting 5s
timeout03 1 TINFO: Test is still running, waiting 4s
timeout03 1 TINFO: Test is still running, waiting 3s
timeout03 1 TINFO: Test is still running, waiting 2s
timeout03 1 TINFO: Test is still running, waiting 1s
timeout03 1 TBROK: Test still running, sending SIGKILL
Killed
(This one based on Joerg's improvement work, better apply this after the two patches)
--- testcases/lib/tst_test.sh | 1 + 1 file changed, 1 insertion(+)
testcases/lib/tst_test.sh +1 -0
@@ -439,6 +439,7 @@ _tst_kill_test() { local i=10 + trap '' INT tst_res TBROK "Test timeouted, sending SIGINT! If you are running on slow machine, try exporting LTP_TIMEOUT_MUL > 1" kill -INT -$pid tst_sleep 100ms
Signature · 1 line
2.30.2
PATCH v3 3/4 lib: ignore SIGINT in _tst_kill_test
PATCH v3 3/4 lib: ignore SIGINT in _tst_kill_test
In reply to Li WangHi Li,
first of all thanks for fixing my patchset and getting it merged.
Quoted 7 lines · Li Wang
We have to guarantee
_tst_kill_testalive for a while to check if the target test eixst or not, so ignore SIGINT in_tst_kill_testis necessary, otherwise it will be stopped by the SIGINT sending by itself.The
timeout03.shverify this mechanism proccess well in output:
Quoted 15 lines
timeout03 1 TBROK: Test timeouted, sending SIGINT! If you are running on slow machine, try exporting
LTP_TIMEOUT_MUL> 1 timeout03 1 TBROK: test interrupted or timed out
timeout03 1 TPASS: test run cleanup after timeout
timeout03 1 TINFO: Test is still running, waiting 10s
timeout03 1 TINFO: Test is still running, waiting 9s
timeout03 1 TINFO: Test is still running, waiting 8s
timeout03 1 TINFO: Test is still running, waiting 7s
timeout03 1 TINFO: Test is still running, waiting 6s
timeout03 1 TINFO: Test is still running, waiting 5s
timeout03 1 TINFO: Test is still running, waiting 4s
timeout03 1 TINFO: Test is still running, waiting 3s
timeout03 1 TINFO: Test is still running, waiting 2s
timeout03 1 TINFO: Test is still running, waiting 1s
timeout03 1 TBROK: Test still running, sending SIGKILL
Killed
At first I did bot understand the problem you found, because I tried with dash, busybox sh and zsh.
All three shells had no problem here. But then I tried with bash and it failed.
I wonder if this is a bug in bash or in the other shells. I guess sending the signal to the whole
process group should also send it to the process running _tst_kill_test.
I did some digging into this while writing this (see conclusion below for results only):
1. All shells have their own implementation of kill (compare <SHELL> -c kill with /usr/bin kill)
2. When replacing "just" kill in the script with /usr/bin/kill, it still only fails in bash.
3. zsh seems to ignore SIGINT, but it can be caught using trap. busybox sh, and dash can't even get it when trapped
4. zsh disables SIGINT by callling "trap '' INT" internally somehow. When resetting SIGINT to default behavior, it is the same as bash.
For zsh this seems to be default behavior for background processes, probably to prevent keyboard interruption by CTRL+C:
zsh -c "trap&"
trap -- '' INT
trap -- '' QUIT
zsh -c "trap"
# No output
To conclude:
- bash does not seem to care about SIGINT delivery to background processes, but can be blocked using trap
- zsh ignores SIGINT for background processes by default, but can be allowed using trap
- dash and busybox sh ignore the signal to background processes, and this cannot be changed with trap
I tried with the following snippets:
<SHELL> -c 'trap "echo trap;" INT; (sleep 2; echo end sub) & sleep 1; kill -INT -$$; echo end main'
<SHELL> -c 'trap "echo trap;" INT; (trap - SIGINT sleep 2; echo end sub) & sleep 1; kill -INT -$$; echo end main'
<SHELL> -c 'trap "echo trap;" INT; (trap "exit" SIGINT sleep 2; echo end sub) & sleep 1; kill -INT -$$; echo end main'
SIGINT handling for child processes is strange. This might have some implication for the shell tests,
because it is possible, that SIGINT is not delivered to all processes and some may reside as orphans.
Since this can happen only in case of timeouts, I guess there is no real Problem.
A possible fix could be using SIGTERM instead of SIGINT. This signal does not seem to have some "intelligent" handling for background processes. I do not know why LTP used SIGINT in the first place. My first thought would have been to use SIGTERM.? It is the way to "politely ask processes to terminate"
J?rg
PATCH v3 3/4 lib: ignore SIGINT in _tst_kill_test
Hi Joerg,
Quoted 67 lines · Joerg Vehlow
Hi Li,
first of all thanks for fixing my patchset and getting it merged.
Quoted 7 lines · Li Wang
We have to guarantee
_tst_kill_testalive for a while to check if the target test eixst or not, so ignore SIGINT in_tst_kill_testis necessary, otherwise it will be stopped by the SIGINT sending by itself.The
timeout03.shverify this mechanism proccess well in output:Quoted 15 lines
timeout03 1 TBROK: Test timeouted, sending SIGINT! If you are running on slow machine, try exporting
LTP_TIMEOUT_MUL> 1 timeout03 1 TBROK: test interrupted or timed out
timeout03 1 TPASS: test run cleanup after timeout
timeout03 1 TINFO: Test is still running, waiting 10s
timeout03 1 TINFO: Test is still running, waiting 9s
timeout03 1 TINFO: Test is still running, waiting 8s
timeout03 1 TINFO: Test is still running, waiting 7s
timeout03 1 TINFO: Test is still running, waiting 6s
timeout03 1 TINFO: Test is still running, waiting 5s
timeout03 1 TINFO: Test is still running, waiting 4s
timeout03 1 TINFO: Test is still running, waiting 3s
timeout03 1 TINFO: Test is still running, waiting 2s
timeout03 1 TINFO: Test is still running, waiting 1s
timeout03 1 TBROK: Test still running, sending SIGKILL
KilledAt first I did bot understand the problem you found, because I tried with dash, busybox sh and zsh.
All three shells had no problem here. But then I tried with bash and it failed.I wonder if this is a bug in bash or in the other shells. I guess sending the signal to the whole
process group should also send it to the process running_tst_kill_test.I did some digging into this while writing this (see conclusion below for results only):
1. All shells have their own implementation of kill (compare <SHELL> -c kill with/usr/binkill)
2. When replacing "just" kill in the script with/usr/bin/kill, it still only fails in bash.
3. zsh seems to ignore SIGINT, but it can be caught using trap. busybox sh, and dash can't even get it when trapped
4. zsh disables SIGINT by callling "trap '' INT" internally somehow. When resetting SIGINT to default behavior, it is the same as bash.For zsh this seems to be default behavior for background processes, probably to prevent keyboard interruption by CTRL+C:
zsh -c "trap&"
trap -- '' INT
trap -- '' QUITzsh -c "trap"
# No outputTo conclude:
- bash does not seem to care about SIGINT delivery to background processes, but can be blocked using trap
- zsh ignores SIGINT for background processes by default, but can be allowed using trap
- dash and busybox sh ignore the signal to background processes, and this cannot be changed with trap
Quoted 8 lines
I tried with the following snippets:
<SHELL> -c 'trap "echo trap;" INT; (sleep 2; echo end sub) & sleep 1; kill -INT -$$; echo end main'
<SHELL> -c 'trap "echo trap;" INT; (trap - SIGINT sleep 2; echo end sub) & sleep 1; kill -INT -$$; echo end main'
<SHELL> -c 'trap "echo trap;" INT; (trap "exit" SIGINT sleep 2; echo end sub) & sleep 1; kill -INT -$$; echo end main'
Thanks for the demos above, it shows the difference clearly.
Quoted 6 lines
SIGINT handling for child processes is strange. This might have some implication for the shell tests,
because it is possible, that SIGINT is not delivered to all processes and some may reside as orphans.
Since this can happen only in case of timeouts, I guess there is no real Problem.
Yes.
Looks like the behaviors on signal 'SIGINT' are not unify in background processes handling for different SHELL. So as you said that using SIGINT seems NOT a good idea to stop the process in timeout.
Quoted 3 lines
A possible fix could be using SIGTERM instead of SIGINT. This signal does not seem to have some "intelligent" handling for background processes.
I agree. Can you make a patch to replace that INT?
(and this is only a timeout issue, so patch merging may be delayed due to LTP new release)
Quoted 3 lines
I do not know why LTP used SIGINT in the first place. My first thought would have been to use SIGTERM. It is the way to "politely ask processes to terminate"
Yes, but that not strange to me, the possible reason is just to stop(ctrl ^c) the LTP test manually for debugging, so we went too far for using SIGINT but forget the original purpose :).
Signature · 2 lines
Regards, Li Wang
PATCH v3 3/4 lib: ignore SIGINT in _tst_kill_test
Hi Li,
Quoted 6 lines
Quoted 2 lines
A possible fix could be using SIGTERM instead of SIGINT. This signal does not seem to have some "intelligent" handling for background processes.
I agree. Can you make a patch to replace that INT?
(and this is only a timeout issue, so patch merging may be delayed due to LTP new release)
I'd like to supply the patch, I've placed it on my todo list. I will probably not finish it before the release,
but since it will probably not be included anyway, it doesn't matter. I do not know why LTP used SIGINT in the first place. My first thought
Quoted 5 lines
Quoted 2 lines
would have been to use SIGTERM. It is the way to "politely ask processes to terminate"
Yes, but that not strange to me, the possible reason is just to stop(ctrl ^c) the LTP test manually for debugging, so we went too far for using SIGINT but forget the original purpose :).
Ok sounds reasonable. The nice think about changing timeout to SIGTERM would be,
that abort using CTRL+C is clearly distinguishable from a timeout.
J?rg
PATCH v3 3/4 lib: ignore SIGINT in _tst_kill_test
In reply to Li WangHi all,
...
Quoted 8 lines
Quoted 7 lines
To conclude:
- bash does not seem to care about SIGINT delivery to background processes, but can be blocked using trap
- zsh ignores SIGINT for background processes by default, but can be allowed using trap
- dash and busybox sh ignore the signal to background processes, and this cannot be changed with trap
Quoted 7 lines
Quoted 7 lines
I tried with the following snippets:
<SHELL> -c 'trap "echo trap;" INT; (sleep 2; echo end sub) & sleep 1; kill -INT -$$; echo end main'
<SHELL> -c 'trap "echo trap;" INT; (trap - SIGINT sleep 2; echo end sub) & sleep 1; kill -INT -$$; echo end main'
<SHELL> -c 'trap "echo trap;" INT; (trap "exit" SIGINT sleep 2; echo end sub) & sleep 1; kill -INT -$$; echo end main'
FYI (you probably know it) SIGINT is a bashism, INT needs to be used.
$ kill -s SIGINT $$ dash: 2: kill: invalid signal number or name: SIGINT
Quoted 9 lines
Thanks for the demos above, it shows the difference clearly.
Quoted 6 lines
SIGINT handling for child processes is strange. This might have some implication for the shell tests,
because it is possible, that SIGINT is not delivered to all processes and some may reside as orphans.
Since this can happen only in case of timeouts, I guess there is no real Problem.
Quoted 5 lines
Yes.
Looks like the behaviors on signal 'SIGINT' are not unify in background processes handling for different SHELL. So as you said that using SIGINT seems NOT a good idea to stop the process in timeout.
Yes, trap looks to be having some portability issues [1] [2]
Quoted 8 lines
Quoted 2 lines
A possible fix could be using SIGTERM instead of SIGINT. This signal does not seem to have some "intelligent" handling for background processes.
I agree. Can you make a patch to replace that INT?
(and this is only a timeout issue, so patch merging may be delayed due to LTP new release)
Quoted 7 lines
Quoted 3 lines
I do not know why LTP used SIGINT in the first place. My first thought would have been to use SIGTERM. It is the way to "politely ask processes to terminate"
Yes, but that not strange to me, the possible reason is just to stop(ctrl ^c) the LTP test manually for debugging, so we went too far for using SIGINT but forget the original purpose :).
I'm not the author, but yes, SIGINT was used for stopping with ctrl^c during debugging.
FYI I tried to use both SIGINT and SIGTERM, but there was some problem. But I guess it was my error. Please *add* SIGTERM (keep SIGINT).
Kind regards,
Petr
[1] https://unix.stackexchange.com/questions/240723/exit-trap-in-dash-vs-ksh-and-bash/240736#240736
[2] https://stackoverflow.com/questions/27012762/is-trap-exit-required-to-execute-in-case-of-sigint-or-sigterm-received
PATCH v3 3/4 lib: ignore SIGINT in _tst_kill_test
Quoted 2 lines
FYI I tried to use both SIGINT and SIGTERM, but there was some problem. But I guess it was my error. Please *add* SIGTERM (keep SIGINT).
Yes, we'd better keep SIGINT for ctrl^c action and use SIGTERM additionally for process terminating.
Does this below (rough solution in my mind) work for you?
(quoted diff) +0 -0
b/lib/newlib_tests/shell/timeout03.sh
index cd548d9a2..f39f5712a 100755
lib/newlib_tests/shell/timeout03.sh +1 -0
@@ -30,6 +30,7 @@ TST_TIMEOUT=1 do_test() { + trap "echo 'Sorry, timeout03 is still alive'" TERM tst_res TINFO "testing killing test after TST_TIMEOUT" sleep 2
testcases/lib/tst_test.sh +1 -3
@@ -21,7 +21,7 @@ export TST_LIB_LOADED=1 . tst_security.sh # default trap function -trap "tst_brk TBROK 'test interrupted or timed out'" INT +trap "tst_brk TBROK 'test interrupted'" INT _tst_do_exit() {
@@ -439,18 +439,18 @@ _tst_kill_test() { local i=10 - trap '' INT - tst_res TBROK "Test timeouted, sending SIGINT! If you are
running on slow machine, try exporting LTP_TIMEOUT_MUL > 1"
- kill -INT -$pid
+ trap '' TERM
+ tst_res TBROK "Test timeouted, sending SIGTERM! If you are
running on slow machine, try exporting LTP_TIMEOUT_MUL > 1"
+ kill -TERM -$pid
tst_sleep 100ms- while kill -0 $pid 2>&1 > /dev/null && [ $i -gt 0 ]; do
+ while kill -0 $pid &>/dev/null && [ $i -gt 0 ]; do
tst_res TINFO "Test is still running, waiting ${i}s"
sleep 1
i=$((i-1))
done- if kill -0 $pid 2>&1 > /dev/null; then
+ if kill -0 $pid &>/dev/null; then
tst_res TBROK "Test still running, sending SIGKILL" kill -KILL -$pid
fi
Signature · 2 lines
Regards, Li Wang
PATCH v3 3/4 lib: ignore SIGINT in _tst_kill_test
Quoted 7 lines
Quoted 2 lines
FYI I tried to use both SIGINT and SIGTERM, but there was some problem. But I guess it was my error. Please *add* SIGTERM (keep SIGINT).
Yes, we'd better keep SIGINT for ctrl^c action and use SIGTERM additionally for process terminating.
Does this below (rough solution in my mind) work for you?
LGTM, but Joerg, Metan, could you please have a look?
lib/newlib_tests/shell/timeout03.sh TST_TIMEOUT=1 · line 33
Quoted 10 lines
lib/newlib_tests/shell/timeout03.sh +1
-0@@ -30,6 +30,7 @@ TST_TIMEOUT=1 do_test() { + trap "echo 'Sorry, timeout03 is still alive'" TERM
Any reason why not use tst_res TINFO? (working on bash).
Quoted 27 lines
lib/newlib_tests/shell/timeout03.sh +0
-0tst_res TINFO "testing killing test after TST_TIMEOUT" sleep 2testcases/lib/tst_test.sh +1
-3@@ -21,7 +21,7 @@ export TST_LIB_LOADED=1 . tst_security.sh # default trap function -trap "tst_brk TBROK 'test interrupted or timed out'" INT +trap "tst_brk TBROK 'test interrupted'" INT _tst_do_exit() {@@ -439,18 +439,18 @@ _tst_kill_test() { local i=10 - trap '' INT - tst_res TBROK "Test timeouted, sending SIGINT! If you arerunning on slow machine, try exporting
LTP_TIMEOUT_MUL> 1"testcases/lib/tst_test.sh +2
-1- kill -INT -$pid + trap '' TERM + tst_res TBROK "Test timeouted, sending SIGTERM! If you arerunning on slow machine, try exporting
LTP_TIMEOUT_MUL> 1"
testcases/lib/tst_test.sh
Quoted 5 lines
testcases/lib/tst_test.sh +2
-1+ kill -TERM -$pid tst_sleep 100ms - while kill -0 $pid 2>&1 > /dev/null && [ $i -gt 0 ]; do + while kill -0 $pid &>/dev/null && [ $i -gt 0 ]; do
FYI: &> is a bashism (we need to keep the original).
testcases/lib/tst_test.sh
Quoted 7 lines
tst_res TINFO "Test is still running, waiting ${i}s" sleep 1 i=$((i-1)) donetestcases/lib/tst_test.sh +1
-1- if kill -0 $pid 2>&1 > /dev/null; then + if kill -0 $pid &>/dev/null; then
And here as well.
Quoted 3 lines
tst_resTBROK "Test still running, sending SIGKILL" kill -KILL -$pid
fi
Kind regards,
Petr
PATCH v3 3/4 lib: ignore SIGINT in _tst_kill_test
Quoted 2 lines · Petr Vorel
Quoted 1 line
Does this below (rough solution in my mind) work for you?
LGTM, but Joerg, Metan, could you please have a look?
Thanks, I wouldn't send a patch until Joerg/Cyril has a review. (Maybe Joerg will have another better solution:)
Quoted 11 lines
lib/newlib_tests/shell/timeout03.sh TST_TIMEOUT=1 · line 33
Quoted 10 lines
lib/newlib_tests/shell/timeout03.sh +1
-0@@ -30,6 +30,7 @@ TST_TIMEOUT=1 do_test() { + trap "echo 'Sorry, timeout03 is still alive'" TERMAny reason why not use
tst_resTINFO? (working on bash).
+1
I just use echo for a quick test, but surely we can replace it with tst_res.
Quoted 3 lines
lib/newlib_tests/shell/timeout03.sh TST_TIMEOUT=1 · line 34
Quoted 2 lines
lib/newlib_tests/shell/timeout03.sh +1
-1- while kill -0 $pid 2>&1 > /dev/null && [ $i -gt 0 ]; do + while kill -0 $pid &>/dev/null && [ $i -gt 0 ]; doFYI: &> is a bashism (we need to keep the original).
I just want the error does not to redirect to standard output. Because with SIGTERM sending, it seems easier to kill all processes, so 'kill -0 $pid' returns "No such process" errors often.
Mayby I should go with: kill -0 $pid >/dev/null 2>&1
e.g.
# ./timeout04.sh timeout04 1 TINFO: timeout per run is 0h 0m 1s timeout04 1 TINFO: Start timeout04 1 TBROK: Test timeouted, sending SIGTERM! If you are running on slow machine, try exporting LTP_TIMEOUT_MUL > 1 Terminated ./../../../testcases/lib/tst_test.sh: line 448: kill: (74911) - No such process ./../../../testcases/lib/tst_test.sh: line 454: kill: (74911) - No such process
Signature · 2 lines
Regards, Li Wang
PATCH v3 3/4 lib: ignore SIGINT in _tst_kill_test
Hi Li, Joerg, Cyril,
Quoted 7 lines
Quoted 2 lines · Petr Vorel
Quoted 1 line
Does this below (rough solution in my mind) work for you?
LGTM, but Joerg, Metan, could you please have a look?
Thanks, I wouldn't send a patch until Joerg/Cyril has a review. (Maybe Joerg will have another better solution:)
Understand.
Quoted 7 lines
Quoted 6 lines
Quoted 6 lines
lib/newlib_tests/shell/timeout03.sh +0
-0@@ -30,6 +30,7 @@ TST_TIMEOUT=1
Quoted 7 lines
Quoted 4 lines
lib/newlib_tests/shell/timeout03.sh TST_TIMEOUT=1 · line 33
Quoted 3 lines
lib/newlib_tests/shell/timeout03.sh +1
-0do_test() { + trap "echo 'Sorry, timeout03 is still alive'" TERMAny reason why not use
tst_resTINFO? (working on bash).+1
I just use echo for a quick test, but surely we can replace it withtst_res.
+1
Quoted 9 lines
Quoted 3 lines
lib/newlib_tests/shell/timeout03.sh TST_TIMEOUT=1 · line 33
Quoted 2 lines
lib/newlib_tests/shell/timeout03.sh +1
-1- while kill -0 $pid 2>&1 > /dev/null && [ $i -gt 0 ]; do + while kill -0 $pid &>/dev/null && [ $i -gt 0 ]; doFYI: &> is a bashism (we need to keep the original).
I just want the error does not to redirect to standard output. Because with SIGTERM sending, it seems easier to kill all processes, so 'kill -0 $pid' returns "No such process" errors often.
Mayby I should go with: kill -0 $pid >/dev/null 2>&1
+1. Now I see it, "2>&1 > /dev/null" is wrong, it must be vice versa.
Kind regards,
Petr
Quoted 10 lines
e.g.
# ./timeout04.sh timeout04 1 TINFO: timeout per run is 0h 0m 1s timeout04 1 TINFO: Start timeout04 1 TBROK: Test timeouted, sending SIGTERM! If you are running on slow machine, try exporting LTP_TIMEOUT_MUL > 1 Terminated ./../../../testcases/lib/tst_test.sh: line 448: kill: (74911) - No such process ./../../../testcases/lib/tst_test.sh: line 454: kill: (74911) - No such process
PATCH v3 3/4 lib: ignore SIGINT in _tst_kill_test
In reply to Li WangHi Li,
Quoted 25 lines · Li Wang
Quoted 2 lines
FYI I tried to use both SIGINT and SIGTERM, but there was some problem. But I guess it was my error. Please *add* SIGTERM (keep SIGINT).
Yes, we'd better keep SIGINT for ctrl^c action and use SIGTERM additionally for process terminating.
Does this below (rough solution in my mind) work for you?
lib/newlib_tests/shell/timeout03.sh +1
-0@@ -30,6 +30,7 @@ TST_TIMEOUT=1 do_test() { + trap "echo 'Sorry, timeout03 is still alive'" TERM tst_res TINFO "testing killing test after TST_TIMEOUT" sleep 2testcases/lib/tst_test.sh +0
-0
testcases/lib/tst_test.sh export TST_LIB_LOADED=1 · line 24
Quoted 5 lines
testcases/lib/tst_test.sh +1
-1@@ -21,7 +21,7 @@ export TST_LIB_LOADED=1 . tst_security.sh # default trap function -trap "tst_brk TBROK 'test interrupted or timed out'" INT +trap "tst_brk TBROK 'test interrupted'" INT
This would require something like
trap "tst_brk TBROK 'test terminated'" TERM
or
trap "_tst_do_exit" TERM
Otherwise the test is terminated very roughly, without executing cleanup, which is probably not a good idea.
But that introduces the next problem: A short deadlock between _tst_kill_test and _tst_cleanup_timer,
because _tst_cleanup_timer waits for the termination of the timeout process and vice versa.
Another problem is, that a SIGTERM originating from some other location could look like a timeout.
I am currently thinking about the following solution, to mitigate most problems:
The timeout process sends SIGUSR1 (or maybe SIGALRM?) only to the main test process and blocks TERM.
The main process can print, that it ran into a timeout, send a sigterm to its processs group (while ignoring TERM itself).
Then it can unset $_tst_setup_timer_pid safely, because it knows it was triggered by the timeout process and execute _tst_do_exit.
If the timeout process does not see the termination of the main process, it can still send SIGKILL to the whole process group.
J?rg
PATCH v3 3/4 lib: ignore SIGINT in _tst_kill_test
Hi Joerg,
Quoted 9 lines
Review
Quoted 2 lines
(quoted diff) +1
-1-trap "tst_brk TBROK 'test interrupted or timed out'" INT +trap "tst_brk TBROK 'test interrupted'" INTThis would require something like
trap "tst_brkTBROK 'test terminated'" TERM
or
trap "_tst_do_exit" TERMOtherwise the test is terminated very roughly, without executing cleanup, which is probably not a good idea.
Yes, seems I didn't realize this needs cleanup as well.
But I'd still suggest keeping SIGINT here for catching Ctrl^C for users :).
Quoted 7 lines
But that introduces the next problem: A short deadlock between
_tst_kill_testand_tst_cleanup_timer,
because_tst_cleanup_timerwaits for the termination of the timeout process and vice versa.
Another problem is, that a SIGTERM originating from some other location could look like a timeout.
Yes, and that's the reason why I didn't trap SIGTERM simply in the main process.
Quoted 11 lines
I am currently thinking about the following solution, to mitigate most problems:
The timeout process sends SIGUSR1 (or maybe SIGALRM?) only to the main test process and blocks TERM.
The main process can print, that it ran into a timeout, send a sigterm to its processs group (while ignoring TERM itself).
Then it can unset $_tst_setup_timer_pid safely, because it knows it was triggered by the timeout process and execute_tst_do_exit.If the timeout process does not see the termination of the main process, it can still send SIGKILL to the whole process group.
It probably will be work but looks a bit confusing since that involves more signals.
In conclusion, I think we maybe have such situations to be solved:
1. SIGINT (Ctrl^C) for terminating the main process and do cleanup correctly before a timeout
2. Test finish normally and retrieves the _tst_timeout_process in the background via SIGTERM(sending by _tst_cleanup_timer)
3. Test timeout occurs and _tst_kill_test sending SIGTERM to terminating all process, and the main process do cleanup work
4. Test timeout occurs but still have process alive after _tst_kill_test sending SIGTERM, then sending SIGKILL to the whole group
So, I'm now thinking can we just introduce a knob(variable) for skipping the _tst_cleanup_timer works in timeout mode, then it will not have a deadlock anymore.
How about:
testcases/lib/tst_test.sh +6 -4
@@ -16,12 +16,14 @@ export TST_COUNT=1 export TST_ITERATIONS=1 export TST_TMPDIR_RHOST=0 export TST_LIB_LOADED=1 +export TST_TIMEOUT_OCCUR=0 . tst_ansi_color.sh . tst_security.sh # default trap function -trap "tst_brk TBROK 'test interrupted or timed out'" INT +trap "tst_brk TBROK 'test interrupted'" INT +trap "TST_TIMEOUT_OCCUR=1; tst_brk TBROK 'test timeouted'" TERM _tst_do_exit() {
@@ -48,7 +50,9 @@ _tst_do_exit() [ "$TST_TMPDIR_RHOST" = 1 ] && tst_cleanup_rhost fi - _tst_cleanup_timer + if ["$TST_TIMEOUT_OCCUR" = 0 ]; then + _tst_cleanup_timer + fi if [ $TST_FAIL -gt 0 ]; then ret=$((ret|1))
@@ -439,18 +443,18 @@ _tst_kill_test() { local i=10 - trap '' INT - tst_res TBROK "Test timeouted, sending SIGINT! If you are
running on slow machine, try exporting LTP_TIMEOUT_MUL > 1"
- kill -INT -$pid
+ trap '' TERM
+ tst_res TBROK "Test timeouted, sending SIGTERM! If you are
running on slow machine, try exporting LTP_TIMEOUT_MUL > 1"
+ kill -TERM -$pid
tst_sleep 100ms- while kill -0 $pid 2>&1 > /dev/null && [ $i -gt 0 ]; do
+ while kill -0 $pid >/dev/null 2>&1 && [ $i -gt 0 ]; do
tst_res TINFO "Test is still running, waiting ${i}s"
sleep 1
i=$((i-1))
done- if kill -0 $pid 2>&1 > /dev/null; then
+ if kill -0 $pid >/dev/null 2>&1; then
tst_res TBROK "Test still running, sending SIGKILL" kill -KILL -$pid
fi
--
Regards,
Li Wang
PATCH v3 3/4 lib: ignore SIGINT in _tst_kill_test
Hi all,
Quoted 11 lines
Hi Joerg,
Quoted 9 lines
testcases/lib/tst_test.sh export TST_COUNT=1 · line 25
Quoted 2 lines
testcases/lib/tst_test.sh +1
-1-trap "tst_brk TBROK 'test interrupted or timed out'" INT +trap "tst_brk TBROK 'test interrupted'" INTThis would require something like
trap "tst_brkTBROK 'test terminated'" TERM
or
trap "_tst_do_exit" TERMOtherwise the test is terminated very roughly, without executing cleanup, which is probably not a good idea.
+1
Quoted 3 lines
Yes, seems I didn't realize this needs cleanup as well.
But I'd still suggest keeping SIGINT here for catching Ctrl^C for users :).
+1
Quoted 38 lines
Quoted 6 lines
But that introduces the next problem: A short deadlock between
_tst_kill_testand_tst_cleanup_timer,
because_tst_cleanup_timerwaits for the termination of the timeout process and vice versa.
Another problem is, that a SIGTERM originating from some other location could look like a timeout.Yes, and that's the reason why I didn't trap SIGTERM simply in the main process.
Quoted 11 lines
I am currently thinking about the following solution, to mitigate most problems:
The timeout process sends SIGUSR1 (or maybe SIGALRM?) only to the main test process and blocks TERM.
The main process can print, that it ran into a timeout, send a sigterm to its processs group (while ignoring TERM itself).
Then it can unset $_tst_setup_timer_pid safely, because it knows it was triggered by the timeout process and execute_tst_do_exit.If the timeout process does not see the termination of the main process, it can still send SIGKILL to the whole process group.
It probably will be work but looks a bit confusing since that involves more signals.
In conclusion, I think we maybe have such situations to be solved:
1. SIGINT (Ctrl^C) for terminating the main process and do cleanup correctly before a timeout
2. Test finish normally and retrieves the_tst_timeout_processin the background viaSIGTERM(sending by _tst_cleanup_timer)
3. Test timeout occurs and_tst_kill_testsending SIGTERM to terminating all process, and the main process do cleanup work
4. Test timeout occurs but still have process alive after_tst_kill_testsending SIGTERM, then sending SIGKILL to the whole group
Quoted 3 lines
So, I'm now thinking can we just introduce a knob(variable) for skipping the
_tst_cleanup_timerworks in timeout mode, then it will not have a deadlock anymore.
+1
Quoted 1 line
How about:
I'm not sure if we're not getting too late for these changes. Because it'll be just on us to test that (community probably have run these changes). But I'd still prefer to fix it.
If we don't fix it, I'd be at least for fixing wrong redirection order (2>&1 > /dev/null) introduced by me in 25ad54dba.
Quoted 45 lines
testcases/lib/tst_test.sh +6
-4@@ -16,12 +16,14 @@ export TST_COUNT=1 export TST_ITERATIONS=1 export TST_TMPDIR_RHOST=0 export TST_LIB_LOADED=1 +export TST_TIMEOUT_OCCUR=0 . tst_ansi_color.sh . tst_security.sh # default trap function -trap "tst_brk TBROK 'test interrupted or timed out'" INT +trap "tst_brk TBROK 'test interrupted'" INT +trap "TST_TIMEOUT_OCCUR=1; tst_brk TBROK 'test timeouted'" TERM _tst_do_exit() {@@ -48,7 +50,9 @@ _tst_do_exit() [ "$TST_TMPDIR_RHOST" = 1 ] && tst_cleanup_rhost fi - _tst_cleanup_timer + if ["$TST_TIMEOUT_OCCUR" = 0 ]; then + _tst_cleanup_timer + fi if [ $TST_FAIL -gt 0 ]; then ret=$((ret|1))@@ -439,18 +443,18 @@ _tst_kill_test() { local i=10 - trap '' INT - tst_res TBROK "Test timeouted, sending SIGINT! If you arerunning on slow machine, try exporting
LTP_TIMEOUT_MUL> 1"testcases/lib/tst_test.sh +2
-1- kill -INT -$pid + trap '' TERM + tst_res TBROK "Test timeouted, sending SIGTERM! If you arerunning on slow machine, try exporting
LTP_TIMEOUT_MUL> 1"testcases/lib/tst_test.sh +2
-1+ kill -TERM -$pid tst_sleep 100ms - while kill -0 $pid 2>&1 > /dev/null && [ $i -gt 0 ]; do + while kill -0 $pid >/dev/null 2>&1 && [ $i -gt 0 ]; do
testcases/lib/tst_test.sh
Quoted 10 lines
testcases/lib/tst_test.sh +1
-1tst_res TINFO "Test is still running, waiting ${i}s" sleep 1 i=$((i-1)) done - if kill -0 $pid 2>&1 > /dev/null; then + if kill -0 $pid >/dev/null 2>&1; then tst_res TBROK "Test still running, sending SIGKILL" kill -KILL -$pid fi
LGTM, I'll try to test it today.
Kind regards,
Petr
PATCH v3 3/4 lib: ignore SIGINT in _tst_kill_test
In reply to Li WangHi,
Quoted 9 lines · Li Wang
Hi Joerg,
Quoted 7 lines
testcases/lib/tst_test.sh export TST_COUNT=1 · line 25
Quoted 2 lines
testcases/lib/tst_test.sh +1
-1-trap "tst_brk TBROK 'test interrupted or timed out'" INT +trap "tst_brk TBROK 'test interrupted'" INTThis would require something like
trap "tst_brkTBROK 'test terminated'" TERM
or
trap "_tst_do_exit" TERM
Quoted 5 lines
Quoted 2 lines
Otherwise the test is terminated very roughly, without executing cleanup, which is probably not a good idea.
Yes, seems I didn't realize this needs cleanup as well.
But I'd still suggest keeping SIGINT here for catching Ctrl^C for users :).
+1, I never intended to remove thi
Quoted 27 lines
Quoted 11 lines
I am currently thinking about the following solution, to mitigate most problems:
The timeout process sends SIGUSR1 (or maybe SIGALRM?) only to the main test process and blocks TERM.
The main process can print, that it ran into a timeout, send a sigterm to its processs group (while ignoring TERM itself).
Then it can unset $_tst_setup_timer_pid safely, because it knows it was triggered by the timeout process and execute_tst_do_exit.If the timeout process does not see the termination of the main process, it can still send SIGKILL to the whole process group.
It probably will be work but looks a bit confusing since that involves more signals.
In conclusion, I think we maybe have such situations to be solved:
1. SIGINT (Ctrl^C) for terminating the main process and do cleanup correctly before a timeout
2. Test finish normally and retrieves the_tst_timeout_processin the background viaSIGTERM(sending by _tst_cleanup_timer)
3. Test timeout occurs and_tst_kill_testsending SIGTERM to terminating all process, and the main process do cleanup work
4. Test timeout occurs but still have process alive after_tst_kill_testsending SIGTERM, then sending SIGKILL to the whole group
Quoted 3 lines
So, I'm now thinking can we just introduce a knob(variable) for skipping the
_tst_cleanup_timerworks in timeout mode, then it will not have a deadlock anymore.
This works of course and is the "simplest" solution, the only thing I do not like about this,
is the fact, that SIGTERM send by something else (e.g. system shoutdown or process manager),
is handled like timeouts are handled and reported as timeout. That's why I suggested introducing
a new signal. But since this is probably rare, I could live without it.
Quoted 8 lines
How about:
testcases/lib/tst_test.sh +0
-0@@ -16,12 +16,14 @@ export TST_COUNT=1 export TST_ITERATIONS=1 export TST_TMPDIR_RHOST=0
testcases/lib/tst_test.sh export TST_COUNT=1 · line 26
Quoted 10 lines
testcases/lib/tst_test.sh +3
-1export TST_LIB_LOADED=1 +export TST_TIMEOUT_OCCUR=0 . tst_ansi_color.sh . tst_security.sh # default trap function -trap "tst_brk TBROK 'test interrupted or timed out'" INT +trap "tst_brk TBROK 'test interrupted'" INT +trap "TST_TIMEOUT_OCCUR=1; tst_brk TBROK 'test timeouted'" TERM
This could also be done by "unset _tst_setup_timer_pid" or '_tst_setup_timer_pid=""'.
I guess even if a new variable is introduced, it should start with an _, because it is supposed to be internal to the framework?
Quoted 24 lines
testcases/lib/tst_test.sh +3
-3_tst_do_exit() {@@ -48,7 +50,9 @@ _tst_do_exit() [ "$TST_TMPDIR_RHOST" = 1 ] && tst_cleanup_rhost fi - _tst_cleanup_timer + if ["$TST_TIMEOUT_OCCUR" = 0 ]; then + _tst_cleanup_timer + fi if [ $TST_FAIL -gt 0 ]; then ret=$((ret|1))@@ -439,18 +443,18 @@ _tst_kill_test() { local i=10 - trap '' INT - tst_res TBROK "Test timeouted, sending SIGINT! If you arerunning on slow machine, try exporting
LTP_TIMEOUT_MUL> 1"testcases/lib/tst_test.sh +2
-1- kill -INT -$pid + trap '' TERM + tst_res TBROK "Test timeouted, sending SIGTERM! If you are
Quoted 1 line
running on slow machine, try exporting
LTP_TIMEOUT_MUL> 1"
If you post this as a patch, can you please fix "timeouted" => "timed out"? There is no word "timeouted" in the english language.
Quoted 18 lines
testcases/lib/tst_test.sh +3
-3+ kill -TERM -$pid tst_sleep 100ms - while kill -0 $pid 2>&1 > /dev/null && [ $i -gt 0 ]; do + while kill -0 $pid >/dev/null 2>&1 && [ $i -gt 0 ]; do tst_res TINFO "Test is still running, waiting ${i}s" sleep 1 i=$((i-1)) done - if kill -0 $pid 2>&1 > /dev/null; then + if kill -0 $pid >/dev/null 2>&1; then tst_res TBROK "Test still running, sending SIGKILL" kill -KILL -$pid fi --
Quoted 2 lines
Regards,
Li Wang
@Petr
I wouldn't recommend getting the fix into the release.
The problem is nothing new and does not fix a "real issue" at the moment, but has the risk of introducing something unexpected.
Fixing the output redirection could be done without a major risk, I guess.
J?rg
PATCH v3 3/4 lib: ignore SIGINT in _tst_kill_test
Hi all,
Quoted 15 lines
Quoted 12 lines
In conclusion, I think we maybe have such situations to be solved:
1. SIGINT (Ctrl^C) for terminating the main process and do cleanup correctly before a timeout
2. Test finish normally and retrieves the_tst_timeout_processin the background viaSIGTERM(sending by _tst_cleanup_timer)
3. Test timeout occurs and_tst_kill_testsending SIGTERM to terminating all process, and the main process do cleanup work
4. Test timeout occurs but still have process alive after_tst_kill_testsending SIGTERM, then sending SIGKILL to the whole groupQuoted 3 lines
So, I'm now thinking can we just introduce a knob(variable) for skipping the
_tst_cleanup_timerworks in timeout mode, then it will not have a deadlock anymore.
Quoted 7 lines
This works of course and is the "simplest" solution, the only thing I do not like about this,
is the fact, that SIGTERM send by something else (e.g. system shoutdown or process manager),
is handled like timeouts are handled and reported as timeout. That's why I suggested introducing
a new signal. But since this is probably rare, I could live without it.
Hmm, it wouldn't handle/report like a time-out if we break with "test terminated"
output for a SIGTERM. If we do
trap "unset _tst_setup_timer_pid; tst_brk TBROK 'test terminated'" TERM
in the main process, system will still send SIGTERM to the _tst_timeout_process when shutting down, and the _tst_kill_test will never be called in that case.
Quoted 16 lines
Quoted 16 lines
How about:
testcases/lib/tst_test.sh +1
-0@@ -16,12 +16,14 @@ export TST_COUNT=1 export TST_ITERATIONS=1 export TST_TMPDIR_RHOST=0 export TST_LIB_LOADED=1 +export TST_TIMEOUT_OCCUR=0 . tst_ansi_color.sh . tst_security.sh
Quoted 6 lines
testcases/lib/tst_test.sh export TST_COUNT=1 · line 26
Quoted 4 lines
testcases/lib/tst_test.sh +2
-1# default trap function -trap "tst_brk TBROK 'test interrupted or timed out'" INT +trap "tst_brk TBROK 'test interrupted'" INT +trap "TST_TIMEOUT_OCCUR=1; tst_brk TBROK 'test timeouted'" TERMThis could also be done by "unset
_tst_setup_timer_pid" or '_tst_setup_timer_pid=""'.
+1, 'unset _tst_setup_timer_pid' is a good idea, sorry I was blind here when reading your previous email:).
Quoted 2 lines
I guess even if a new variable is introduced, it should start with an _, because it is supposed to be internal to the framework?
Yes, but let's go with "unset _tst_setup_timer_pid" but not introduce a new variable.
Quoted 20 lines
Quoted 20 lines
testcases/lib/tst_test.sh +3
-1_tst_do_exit() {@@ -48,7 +50,9 @@ _tst_do_exit() [ "$TST_TMPDIR_RHOST" = 1 ] && tst_cleanup_rhost fi - _tst_cleanup_timer + if ["$TST_TIMEOUT_OCCUR" = 0 ]; then + _tst_cleanup_timer + fi if [ $TST_FAIL -gt 0 ]; then ret=$((ret|1))@@ -439,18 +443,18 @@ _tst_kill_test() { local i=10
Quoted 9 lines
Quoted 7 lines
testcases/lib/tst_test.sh +0
-2- trap '' INT - tst_res TBROK "Test timeouted, sending SIGINT! If you arerunning on slow machine, try exporting
LTP_TIMEOUT_MUL> 1"testcases/lib/tst_test.sh +2
-1- kill -INT -$pid + trap '' TERM + tst_res TBROK "Test timeouted, sending SIGTERM! If you arerunning on slow machine, try exporting
LTP_TIMEOUT_MUL> 1"If you post this as a patch, can you please fix "timeouted" => "timed out"? There is no word "timeouted" in the english language.
Sure. Thanks for your strict attitude on syntax.
Quoted 5 lines
@Petr
I wouldn't recommend getting the fix into the release.
The problem is nothing new and does not fix a "real issue" at the moment, but has the risk of introducing something unexpected.
Fixing the output redirection could be done without a major risk, I guess.
I will split the fix into two-part, one for errors redirection, another for SIGTERM using.
Thanks for your review!
--
Regards,
Li Wang