waitid breaks telnet

PATCH In review Joe Korty 4 participants 12 messages Dec 1, 2004 – Dec 2, 2004

Subsystems (3)

MM core Scheduler exec and binfmt

lore

Share & tools

lore

Joe Korty

author

2004-12-01 03:56 UTC #0 lore raw

[ 2nd send, this one from my home email account...]

telnet no longer works:

    # chkconfig telnet on
    # telnet localhost
    Trying 127.0.0.1...
    Connected to localhost (127.0.0.1).
    Escape character is '^]'.
    Red Hat Enterprise Linux WS release 3 (Taroon Update 2)
    Kernel 2.6.10-rc2 on an i686
    Connection closed by foreign host.

A bsearch placed the bug between 2.6.9-rc1-bk[78], another bsearch on the changesets showed the problem is caused by this patch:

    [email protected][torvalds]|ChangeSet|20040831173525|30767

[PATCH] waitid system call

My guess is, something about the new wait4(2) wrapper is causing the telnet daemon to declare success before its child, /bin/login, exits.

Joe

[PS: this email may not get through, our email servers changed recently and I have been having problems. Roland, please ACK me privately as soon as you see this. Thanks]

Andrew Morton

2004-12-01 04:28 UTC #1 lore raw

Quoted 18 lines · Joe Korty

[ 2nd send, this one from my home email account...]
telnet no longer works:
     # chkconfig telnet on
     # telnet localhost
     Trying 127.0.0.1...
     Connected to localhost (127.0.0.1).
     Escape character is '^]'.
     Red Hat Enterprise Linux WS release 3 (Taroon Update 2)
     Kernel 2.6.10-rc2 on an i686
     Connection closed by foreign host.
A bsearch placed the bug between 2.6.9-rc1-bk[78], another bsearch on the changesets showed the problem is caused by this patch:

Quoted 6 lines

     [email protected][torvalds]|ChangeSet|20040831173525|30767
[PATCH] waitid system call
My guess is, something about the new wait4(2) wrapper is causing the telnet daemon to declare success before its child, /bin/login, exits.

I can reproduce this on 2.6.10-rc2, but it seems to have been fixed in more recent kernels. However I cannot think of anything which we did which would have fixed this.

Joe Korty

author

2004-12-01 13:32 UTC #2 lore raw

Quoted 22 lines · Andrew Morton

Quoted 21 lines · Joe Korty
[ 2nd send, this one from my home email account...]
telnet no longer works:
     # chkconfig telnet on
     # telnet localhost
     Trying 127.0.0.1...
     Connected to localhost (127.0.0.1).
     Escape character is '^]'.
     Red Hat Enterprise Linux WS release 3 (Taroon Update 2)
     Kernel 2.6.10-rc2 on an i686
     Connection closed by foreign host.
A bsearch placed the bug between 2.6.9-rc1-bk[78], another bsearch on the changesets showed the problem is caused by this patch:
     [email protected][torvalds]|ChangeSet|20040831173525|30767
[PATCH] waitid system call

Quoted 7 lines

Quoted 3 lines
My guess is, something about the new wait4(2) wrapper is causing the telnet daemon to declare success before its child, /bin/login, exits.
I can reproduce this on 2.6.10-rc2, but it seems to have been fixed in more recent kernels. However I cannot think of anything which we did which would have fixed this.

I was able to reproduce it with the day-before-yesterday''s bitkeeper tree.

My boss sees broken kernels work once in a while. I myself have never been able to get a broken kernel to work. The problem may be a race.

Joe

Roland McGrath

2004-12-01 19:21 UTC #3 lore raw

In reply to Andrew Morton

I've had no luck reproducing that, so there isn't much I can do. The last time someone thought the waitid change broke something random, it was the perturbation of the compiled code vs the issue that the kernel's assembly code doesn't follow the same calling conventions the compiler expects.

Andrew Morton

2004-12-01 19:42 UTC #4 lore raw

Quoted 2 lines · Roland McGrath

I've had no luck reproducing that, so there isn't much I can do.

Did you try bare 2.6.10-rc2?

Quoted 4 lines

The last
time someone thought the waitid change broke something random, it was the perturbation of the compiled code vs the issue that the kernel's assembly code doesn't follow the same calling conventions the compiler expects.

Could be that, but I was able to reproduce it on 2.6.10-rc2 with gcc-2.95.4, with which -mregparm is disabled.

Still. It would be interesting if Joe could retest with CONFIG_REGPARM=n?

Joe Korty

2004-12-01 22:30 UTC #5 lore raw

Quoted 11 lines · Andrew Morton

Quoted 2 lines · Roland McGrath
I've had no luck reproducing that, so there isn't much I can do.
Did you try bare 2.6.10-rc2?
Quoted 4 lines
The last
time someone thought the waitid change broke something random, it was the perturbation of the compiled code vs the issue that the kernel's assembly code doesn't follow the same calling conventions the compiler expects.

Quoted 4 lines

Could be that, but I was able to reproduce it on 2.6.10-rc2 with gcc-2.95.4, with which -mregparm is disabled.
Still. It would be interesting if Joe could retest with CONFIG_REGPARM=n?

CONFIG_REGPARM is not set in all of my kernels (just verified). Joe

Joe Korty

2004-12-01 22:49 UTC #6 lore raw

Quoted 12 lines · Joe Korty

Quoted 11 lines · Andrew Morton
Quoted 2 lines · Roland McGrath
I've had no luck reproducing that, so there isn't much I can do.
Did you try bare 2.6.10-rc2?
Quoted 4 lines
The last
time someone thought the waitid change broke something random, it was the perturbation of the compiled code vs the issue that the kernel's assembly code doesn't follow the same calling conventions the compiler expects.

Quoted 6 lines

Quoted 4 lines
Could be that, but I was able to reproduce it on 2.6.10-rc2 with gcc-2.95.4, with which -mregparm is disabled.
Still. It would be interesting if Joe could retest with CONFIG_REGPARM=n?
CONFIG_REGPARM is not set in all of my kernels (just verified).

More info: I exclusively use CONFIG_SMP and CONFIG_PREEMPT. If it is a race either or both of these is likely to be involved.

Joe

Joe Korty

2004-12-01 23:22 UTC #7 lore raw

Quoted 16 lines · Joe Korty

Quoted 15 lines · Joe Korty
Quoted 14 lines · Andrew Morton
Quoted 2 lines · Roland McGrath
I've had no luck reproducing that, so there isn't much I can do.
Did you try bare 2.6.10-rc2?
Quoted 4 lines
The last
time someone thought the waitid change broke something random, it was the perturbation of the compiled code vs the issue that the kernel's assembly code doesn't follow the same calling conventions the compiler expects.
Could be that, but I was able to reproduce it on 2.6.10-rc2 with gcc-2.95.4, with which -mregparm is disabled.

Quoted 7 lines

Quoted 3 lines
Quoted 1 line
Still. It would be interesting if Joe could retest with CONFIG_REGPARM=n?
CONFIG_REGPARM is not set in all of my kernels (just verified).
More info: I exclusively use CONFIG_SMP and CONFIG_PREEMPT. If it is a race either or both of these is likely to be involved.

Ok, I rebuilt 2.6.9 with CONFIG_PREEMPT=n and telnet failed the one time I tried it.

Then I built with CONFIG_PREEMPT=n and CONFIG=SMP=n and the first telnet attempt succeeded. I then tried six more telnet attempts, two of those failed and the rest succeeded.

Since my earlier testing usually was of only 1 (sometimes
2) telnet attempts per boot, they too may have had some ratio of success/failure other than 100% or 0%.

Joe

Roland McGrath

2004-12-02 00:06 UTC #8 lore raw

Since I can only manage so far to see this once in a blue moon, and you can produce it frequently, it would be helpful if you can diagnose the problem some. That is, figure out exactly what wrong results from a wait* call is at fault.

Thanks,
Roland

Joe Korty

2004-12-02 17:55 UTC #9 lore raw

Quoted 4 lines · Roland McGrath

Since I can only manage so far to see this once in a blue moon, and you can produce it frequently, it would be helpful if you can diagnose the problem some. That is, figure out exactly what wrong results from a wait* call is at fault.

Hi Roland,
I've been playing with this most of the morning, finally got strace attached to the telnet daemon, but it did me no good .. everything works when straced.

My technique was to replace /usr/sbin/in.telnetd with a script that invokes the original binary under strace:

	# cd /usr/sbin
	# mv in.telnetd in.telnet.d.orig
	# cat <<EOF >in.telnetd
	/usr/bin/strace -ff -o /tmp/telnet.log.$$ /usr/sbin/in.telnetd.orig "$@"
	EOF
	# chmod 755 in.telnetd

Earlier this morning I systematically repeated my earlier, haphazard experiments. I built three kernels from two sources: the first source was the pure 2.6.7-rc1-bk7 tree, the second the same tree with the suspect waitid patch applied. From these I built various kernels with and without SMP and PREEMPT and ran at least seven 'telnet' tests on each. The results:

kernel smp preempt | 1 2 3 4 5 6 7 8 9
======================================================

   bk7          Y   Y       | g g g g g g g
   bk7+waitid   Y   Y       | F F F F F F F
   bk7+waitid   N   N       | F g F F g g g F g

Joe Korty

2004-12-02 22:40 UTC #10 lore raw

PATCH 1/1 fix uninitialized variable in waitid(2)

Specify an initial value signal_struct's field stop_state whenever a signal_struct variable is created.

Bug was discovered through the occasional failure of telnet(1) to connect.

Signed-off-by Joe Korty <[email protected]>

kernel/fork.c +1 -0

@@ -733,6 +733,7 @@
 	sig->group_exit_code = 0;
 	sig->group_exit_task = NULL;
 	sig->group_stop_count = 0;
+	sig->stop_state = 0;
 	sig->curr_target = NULL;
 	init_sigpending(&sig->shared_pending);
 	INIT_LIST_HEAD(&sig->posix_timers);

Andrew Morton

2004-12-02 22:47 UTC #11 lore raw

Re: [PATCH] fix uninitialized variable in waitid(2)

Quoted 3 lines · Joe Korty

Specify an initial value signal_struct's field stop_state whenever a signal_struct variable is created.

whew. Thanks.