stallout race between SIGCONT and SIGSTOP
PATCH 1/1 stallout race between SIGCONT and SIGSTOP
Since 2.6.25-git16, the Open POSIX Test Suite test sigaction/10-1 on occasion stalls out. A ^C breaks the test out of the stall.
To see the problem, one must run the test in a loop. The stallout happens anywhere from 3 to approximately 60 iterations. To make the test runtime more bearable, I've been using a custom version that is 8x faster than the original, s/sleep/usleep/g + new sleep constants.
The test in essence does 10 SIGSTOPs and SIGCONTs, interleaved, with a short delay between each SIGSTOP and SIGCONT, but none (other than the small delay of a printf) between each SIGCONT and SIGSTOP:
for(i=0; i<10; i++) {
printf("--> Sending SIGSTOP #%d\n", i);
kill (pid, SIGSTOP);
usleep(125000);
printf("--> Sending SIGCONT #%d\n", i);
kill (pid, SIGCONT);
// usleep(125000); /* this is missing from the real 10-1 */
}When the above commented-out usleep is enabled, the stallout disappears. If instead of adding a usleep, the printf's are removed, the test stalls out immediately. Therefore the problem has something to do with a SIGSTOP being issued 'too soon' after the issuance of a SIGCONT.
Bisection shows that the problem was introduced by
commit e442055193e4584218006e616c9bdce0c5e9ae5c
Author: Oleg Nesterov <[email protected]>
Date: Wed Apr 30 00:52:44 2008 -0700This commit adds code that solves serious race problems by deferring the actual processing of SIGSTOP and SIGCONT to a later time. I suspect it is this deferring that is making SIGCONT sensitive to a SIGSTOP coming in too close on its heels.
The following patch, not to be considered seriously, lends credence to that theory. It reverts a bit of the above commit, forcing SIGCONT (but not SIGSTOP) to be processed immediately. With this patch, I achieved some 1,400 runs before manually stopping the test.
kernel/signal.c +6 -0
@@ -598,6 +598,8 @@return security_task_kill (t, info, sig,0 ); } +static void do_notify_parent_cldstop (struct task_struct *tsk,int why); +/* * Handle magic process-wide effects of stop/continue signals. Unlike * the signal actions, these happen immediately at signal-generation
@@ -682,6 +684,10 @@ signal->flags = why |SIGNAL_STOP_CONTINUED ; signal->group_stop_count =0 ; signal->group_exit_code =0 ; + signal->flags &= ~SIGNAL_CLD_MASK ; +spin_unlock (&p->sighand->siglock); +do_notify_parent_cldstop (p,CLD_CONTINUED ); +spin_lock (&p->sighand->siglock); }else {/* * We are not stopped, but there could be a stop
Sorry! I have to run avay right now, and I will be completely offline tomorrow. I'll return on Thursday.
Quoted 22 lines · Joe Korty
Since 2.6.25-git16, the Open POSIX Test Suite test sigaction/10-1 on occasion stalls out. A ^C breaks the test out of the stall.
To see the problem, one must run the test in a loop. The stallout happens anywhere from 3 to approximately 60 iterations. To make the test runtime more bearable, I've been using a custom version that is 8x faster than the original, s/sleep/usleep/g + new sleep constants.
The test in essence does 10 SIGSTOPs and SIGCONTs, interleaved, with a short delay between each SIGSTOP and SIGCONT, but none (other than the small delay of a printf) between each SIGCONT and SIGSTOP:
for(i=0; i<10; i++) { printf("--> Sending SIGSTOP #%d\n", i); kill (pid, SIGSTOP); usleep(125000); printf("--> Sending SIGCONT #%d\n", i); kill (pid, SIGCONT); // usleep(125000); /* this is missing from the real 10-1 */ }
Quoted 3 lines
When the above commented-out usleep is enabled, the stallout disappears. If instead of adding a usleep, the printf's are removed, the test stalls out immediately.
Could you clarify? Do you mean that the task hangs in sys_kill() ?
Better yet, to avoid a possible confusion, could you please send me the (modified) source code to re-produce the stall ?
Quoted 9 lines
Therefore the problem has something to do with a SIGSTOP being issued 'too soon' after the issuance of a SIGCONT.
Bisection shows that the problem was introduced by
commit e442055193e4584218006e616c9bdce0c5e9ae5c Author: Oleg Nesterov <[email protected]> Date: Wed Apr 30 00:52:44 2008 -0700
Quoted 6 lines
This commit adds code that solves serious race problems by deferring the actual processing of SIGSTOP and SIGCONT to a later time. I suspect it is this deferring that is making SIGCONT sensitive to a SIGSTOP coming in too close on its heels.
The following patch, not to be considered seriously,
Yes, the patch is not for production, but thanks a lot! I am sure it will help to diagnose the problem.
Thanks Joe!
Oleg.
Quoted 2 lines · Oleg Nesterov
I will be completely offline tomorrow.
Cancelled...
Quoted 2 lines
Better yet, to avoid a possible confusion, could you please send me the (modified) source code to re-produce the stall ?
Thanks Joe for the test-case and for your comments.
Joe says:
Quoted 2 lines
So it looks like the test is in error, not the kernel.
and I am happy to agree.
I think sigaction/10-1.c should be fixed, please see the patch below.
In essence the code does:
for (1 .. 10) {
kill(pid, SIGSTOP);
wait_until_we_receive_CLD_STOPPED(); // <---- HANGS
kill(pid, SIGCONT);
}We can look at the code from the different angle, this loop does
kill(pid, SIGCONT); kill(pid, SIGSTOP); wait_until_we_receive_CLD_STOPPED(); // <---- HANGS
and it hangs because CLD_CONTINUED (which is ignored by the SIGCHLD handler) masks CLD_STOPPED.
And yes, the mentioned patch makes the difference. To clarify, it does not defer the "result" of SIGCONT, it only defers the sending of SIGCHLD with .si_code == CLD_CONTINUED.
So, what happens is:
kill(SIGCONT) does its job, CLD_CONTINUED will be sent a bit later.
kill(SIGSTOP) starts another group-stop.
the child sends CLD_CONTINUED, and before the parent deques the signal, it stops and sends CLD_STOPPED.
Now, since SIGCHLD is a non-rt signal, the second SIGCHLD is lost, and wait_until_we_receive_CLD_STOPPED() hangs.
I did the test patch to be sure:
--- 26-rc2/kernel/signal.c~ 2008-09-20 20:37:52.000000000 +0400 +++ 26-rc2/kernel/signal.c 2008-09-24 18:43:34.000000000 +0400 @@ -808,7 +808,7 @@ static int send_signal(int sig, struct s * exactly one non-rt signal, so that we can get more * detailed information about the cause of the signal. */ - if (legacy_queue(pending, sig)) + if (sig != SIGCHLD && legacy_queue(pending, sig)) return 0; /* * fast-pathed signals for kernel-internal things like SIGSTOP
and now your test-case doesn't hang.
Oleg.
posixtestsuite/conformance/interfaces/sigaction/10-1.c renamed from posixtestsuite/conformance/interfaces/sigaction/10-1.c~ +39 -19
@@ -19,19 +19,41 @@#define NUMSTOPS 10int child_stopped =0 ; -int waiting =1 ; +int child_continued =0 ; +int notification; -void handler (int signo,siginfo_t *info,void *context) +void handler (int signo,siginfo_t *info,void *context) { -if (info && info->si_code ==CLD_STOPPED ) { +if (!info) +return ; + + notification = info->si_code; + +switch (notification) { +case CLD_STOPPED :printf ("Child has been stopped\n" ); - waiting =0 ; child_stopped++; +break ; +case CLD_CONTINUED : +printf ("Child has been continued\n" ); + child_continued++; +break ; } } +void wait_for_notification (int val) +{ +struct timeval tv; + +while (notification != val) { + tv.tv_sec =1 ; + tv.tv_usec =0 ; +if (!select (0 ,NULL ,NULL ,NULL , &tv)) +break ; + } +} -int main () +int main (void ) {pid_t pid;struct sigaction act;
@@ -40,7 +62,7 @@ int main() act.sa_sigaction = handler; act.sa_flags =SA_SIGINFO ;sigemptyset (&act.sa_mask); -sigaction (SIGCHLD , &act,0 ); +sigaction (SIGCHLD , &act,0 );if ((pid =fork ()) ==0 ) {/* child */
@@ -54,37 +76,36 @@ int main()return 0 ; }else {/* parent */ -int s; +int s;int i;for (i =0 ; i <NUMSTOPS ; i++) { - waiting =1 ; -printf ("--> Sending SIGSTOP\n" ); + notification =0 ;kill (pid,SIGSTOP );/* Don't let the kernel optimize away queued SIGSTOP/SIGCONT signals. */ -while (waiting) { - tv.tv_sec =1 ; - tv.tv_usec =0 ; -if (!select (0 ,NULL ,NULL ,NULL , &tv)) -break ; - } - +wait_for_notification (CLD_STOPPED );printf ("--> Sending SIGCONT\n" ); + notification =0 ;kill (pid,SIGCONT ); +/* +SIGCHLD doesn't queue, make sure CLD_CONTINUED +doesn't mask the next CLD_STOPPED +*/ +wait_for_notification (CLD_CONTINUED ); } - +/* POSIX specifies default action to be abnormal termination */ kill (pid,SIGHUP );waitpid (pid, &s,0 ); } -if (child_stopped ==NUMSTOPS ) { +if (child_stopped ==NUMSTOPS && child_continued ==NUMSTOPS ) {printf ("Test PASSED\n" );return 0 ; }
@@ -92,4 +113,3 @@ int main()printf ("Test FAILED\n" );return -1 ; } -
Quoted 5 lines · Oleg Nesterov
Joe says:
Quoted 1 line
So it looks like the test is in error, not the kernel.
and I am happy to agree.
I think sigaction/10-1.c should be fixed, please see the patch below.
A year or two ago I sent to Intel some OpenPosixTestSuite fixes, and they were accepted. Send it in (to the people listed in the comments at the front of the .c file), hopefully they are still at Intel.
Quoted 17 lines
Now, since SIGCHLD is a non-rt signal, the second SIGCHLD is lost, and
wait_until_we_receive_CLD_STOPPED()hangs.I did the test patch to be sure:
--- 26-rc2/kernel/signal.c~ 2008-09-20 20:37:52.000000000 +0400 +++ 26-rc2/kernel/signal.c 2008-09-24 18:43:34.000000000 +0400 @@ -808,7 +808,7 @@ static int send_signal(int sig, struct s * exactly one non-rt signal, so that we can get more * detailed information about the cause of the signal. */ - if (legacy_queue(pending, sig)) + if (sig != SIGCHLD && legacy_queue(pending, sig)) return 0; /* * fast-pathed signals for kernel-internal things like SIGSTOP
Quoted 1 line
and now your test-case doesn't hang.
Very interesting! I am not sure this is Posix conformant, as Posix seems to say that posting a SIGSTOP or SIGCHLD clears out all pending SIGSTOPs or SIGCHLDs, so queueing the SIGCHLD might violate the standard. Still it might be workable as it would be hard to see, from userspace, any behavioral difference between queueing (possibly illegal) and synchronous operation (which seems legal), both of which service all SIGCONTs and SIGSTOPs without loss.
Joe
Quoted 11 lines · Joe Korty
Quoted 5 lines · Oleg Nesterov
Joe says:
Quoted 1 line
So it looks like the test is in error, not the kernel.
and I am happy to agree.
I think sigaction/10-1.c should be fixed, please see the patch below.A year or two ago I sent to Intel some OpenPosixTestSuite fixes, and they were accepted. Send it in (to the people listed in the comments at the front of the .c file), hopefully they are still at Intel.
OK, thanks, will do.
Quoted 14 lines
Quoted 14 lines
I did the test patch to be sure:
--- 26-rc2/kernel/signal.c~ 2008-09-20 20:37:52.000000000 +0400 +++ 26-rc2/kernel/signal.c 2008-09-24 18:43:34.000000000 +0400 @@ -808,7 +808,7 @@ static int send_signal(int sig, struct s * exactly one non-rt signal, so that we can get more * detailed information about the cause of the signal. */ - if (legacy_queue(pending, sig)) + if (sig != SIGCHLD && legacy_queue(pending, sig)) return 0; /* * fast-pathed signals for kernel-internal things like SIGSTOP
Quoted 3 lines
Quoted 1 line
and now your test-case doesn't hang.
Very interesting! I am not sure this is Posix conformant,
No, no, the patch is of course wrong, I did it only to check my understanding.
Quoted 3 lines
as Posix
seems to say that posting a SIGSTOP or SIGCHLD clears out all pending SIGSTOPs or SIGCHLDs,
Hmm. Are you sure?
Anyway, this is not what Linux does. If a non-rt signal is pending, the next signal with the same number is silently ignored. SIGCHLD too.
Quoted 1 line
Still it might be workable
Confused. Do you agree the kernel is not buggy?
To clarify, none of SIGCONTs/SIGSTOPs is lost. But the test-case assumes that it must always receive SIGCHLD + CLD_STOPPED. This is not true because SIGCHLD is not queueable, and we have another "stream" of SIGCHLDs which carry CLD_CONTINUED.
For example, the "opposite" code
kill(SIGSTOP); kill(SIGCONT); wait_for_CLD_CONTINUED();
was always wrong, but
kill(SIGCONT); kill(SIGSTOP); wait_for_CLD_STOPPED();
happened to work before that commit. But please note that it is wrong anyway. For example, if we have another sub-thread, we can miss CLD_STOPPED even without the commit which changed the timing.
Oleg.
Quoted 1 line · Oleg Nesterov
Confused. Do you agree the kernel is not buggy?
I agree with you that what the kernel is doing now is correct. Joe
Quoted 5 lines · Joe Korty
Quoted 1 line · Oleg Nesterov
Confused. Do you agree the kernel is not buggy?
I agree with you that what the kernel is doing now is correct.
Great. Thanks a lot for the report and discussion! And thanks for your analysis (sent privately), the problem was completely explained by you.
BTW, I forgot to show the test-case you sent me privately, perhaps someone from CC might want to look... It really helped.
Oleg.
#include <signal.h>
#include <stdio.h>
#include <stdlib.h>
#include <sys/select.h>
#include <sys/wait.h>
#include <sys/time.h>
#include <sys/types.h>
#include <unistd.h>
#define NUMSTOPS 10
int child_stopped = 0;
int waiting = 1;
void handler(int signo, siginfo_t *info, void *context)
{
if (info && info->si_code == CLD_STOPPED) {
printf("Child has been stopped\n");
waiting = 0;
child_stopped++;
}
}int main(void)
{
pid_t pid;
struct sigaction act;
struct timeval tv;
act.sa_sigaction = handler;
act.sa_flags = SA_SIGINFO;
sigemptyset(&act.sa_mask);
sigaction(SIGCHLD, &act, 0);
setbuf(stdout, NULL);
printf("%d SIGSTOP/SIGCONT combinations to be sent.\n", NUMSTOPS);
if ((pid = fork()) == 0) {
/* child */
while(1) {
/* wait forever, or until we are
interrupted by a signal */
tv.tv_sec = 0;
tv.tv_usec = 0;
select(0, NULL, NULL, NULL, &tv);
}
return 0;
} else {
/* parent */
int s;
int i;
usleep(12000);
for (i = 0; i < NUMSTOPS; i++) {
waiting = 1;
printf("--> Sending SIGSTOP #%d\n", i);
kill(pid, SIGSTOP);
/*
Don't let the kernel optimize away queued
SIGSTOP/SIGCONT signals.
*/
while (waiting)
usleep(12000);
printf("--> Sending SIGCONT\n");
kill(pid, SIGCONT);
// usleep(12000);
}
/* POSIX specifies default action to be abnormal termination */
usleep(12000);
kill(pid, SIGKILL);
waitpid(pid, &s, 0);
}
if (child_stopped == NUMSTOPS) {
printf("Test PASSED\n");
return 0;
}
printf("Test FAILED\n");
return -1;
}