2.6.x Fork Problem?

All,
We've got an application (we'll call it AppB) based on apache and a
proprietary module that we push to other Linux boxes. The way we do this is first push a proprietary executable (we'll call it AppA) to the Linux box via rexec or some other mechanism. We then send messages to AppA and tell it to pull AppB from some place. AppA will pull AppB and run its installer, and then starts AppB via a shell script. So process wise, we have a tree like this:

init

  |
  |-AppA
  |
  |-AppB

Once AppB is running, the apache module will periodically call fork and one of the execs to execute some other process. AppB will execute code like this to fork the process:

pid = fork();
switch (pid)
{
case -1:
blah; /* big trouble */
break;
case 0: /* Child */

      blah;
      blah;
      exec(some command here)
      blah; /* If we get here, we're in big trouble. */
      break;
   default: /* Parent */
      pid2 = waitpid(pid, &status, 0);
      if (pid2 == -1)
      {
         blah;  /* check out errno */
      }
}

On 2.4.x, this code works great. But now with 2.6.x (stock SuSE 9.1 and FC2 kernels, as well as home grown kernels), waitpid will always return -1, and errno will be 10 (ECHILD, No child processes). We know for a fact that the fork and exec calls succeed due to debug print outs. Here's the real kicker. We only see this problem when we first push the software and start it. If we push the software and start it, stop it and start it again, then allow it to try the fork/exec again, it succeeds. We have the same problem if we bundle the entire application (AppA and AppB) into an installer and use that to install it on a machine. The installer will start the application, and the fork/exec calls will fail with ECHILD. If we then stop the App and start it again, everything is fine.

I've widdled the code down to just do this:

pid = fork();
switch (pid)
{
case -1:
blah; /* big trouble */
break;
case 0: /* Child */

      exit(1);
      break;
   default: /* Parent */
      pid2 = waitpid(pid, &status, 0);
      if (pid2 == -1)
      {
         blah;  /* check out errno */
      }
}

and I get the same results, so I now the exec has nothing to do with it.

Anyone have any ideas on why this would happen?

Thanks,

Torin Ford
Venturi Technology Partners

Well I see two oddities in the example. You call exit() not _exit() so the child will duplicate various queued stdio of the parent. Doesn't seem to be relevant however.

Secondly and I suspect of importance you don't do anything with SIGCLD so you are inheriting a random status. If the child signal is being ignored then it will be cleared automatically. In that situation your code functionality depends solely upon which thread runs first.

Yup - the parent process executed waitpid before the child process finished the setup. This can happen in a multi-cpu environment or even a single, if the scheduler puts the parent process higher than the child in the queue.

I seem to remember there was a change in the fork handling in 2.5 that allowed this to happen. It does allow for faster server response since the parent (assumed to be a server) can spin on the socket requests and fork handler servers without a delay between the fork and re-examining the incoming request socket.

The way I've been doing the delay is a signal handler (SIGCHLD) to catch the signal, then a "while( 0 < (pid = waitpid(-1,&stat, WNOHANG)))..." loop to catch any/all children. The parent process ends up sleeping in a select until the signal occurs (and the handler executes). Then resumes.

This way the parent process is waiting for ANY child to terminate, not just the one specified, and it waits for a signal - which can't be generated until the/a process fully exists and terminates.

ugh! I can follow the rationale for SMP.

But wouldn't this kind of behavior actually break most real world programs?

Frank

When fork() returns with a pid. The pid is valid. It cannot be that the child doesn't exist yet. This would, as you said, break everything. What seems to happening is the child calls exit() before the parent gets the CPU. The child should wait in exit() until somebody claims its status.

There has never been any guarantee that the child gets the CPU sooner than the parent. If the parent and child need to synchronize things, there are lots of ways to do it.

In the above code there is something missing. in the code shown, the child __will__ wait in exit() until somebody claims its status. However, the child probably did a setsid(), becoming a process-leader or the parent set up a SIGCHLD handler before the fork. In these cases, the exit() will quickly exit because somebody will claim the exit status.

So, by the time the parent gets the CPU, the child is long gone. The solution is to use the default SIGCHLD handler if the parent expects to get the child's status and for the child to not execute setsid(), which will allow init to reap its status.

Cheers,
Dick Johnson
Penguin : Linux version 2.4.26 on an i686 machine (5570.56 BogoMips).
Note 96.31% of all statistics are fiction.

AFAIK a child doing setsid() has no effect whatsoever on any wait*() done by the parent. It just sets a new session leader.

But SIGCHLD set to SIG_IGN instead of SIG_DFL is a perfect explanation. Rereading alan's reply I suddenly got it: "random status" didn't refer to the &status arg but to the signal status. SIG_IGN is inherited I guess so a

	signal(SIGCHLD, SIG_DFL);

once before the fork() should fix it. Hmm, so actually our parent should have reset SIGCHLD before exec'ing this code. This could cause more problems.

Frank

Thanks to everyone for their responses. Alan and others were correct in that we needed to change the signal handler for SIGCHLD. If I remember correctly, apache changes the signal handler for SIGCHLD, so we would have inherited those signal handlers from apache. Adding a signal(SIGCHLD, SIG_DFL) before the fork fixed the problem for us.

Torin Ford
Venturi Technology Partners

Such a behaviour would not be rational, so we don't do anything like that.