Skip to contentSkip to conversation

2.6.25.3: su gets stuck for root

DISCUSSION Harald Dunkel 5 participants 16 messages May 13, 2008 – May 20, 2008

lore
Share & tools
lore
#0 lore raw

Hi folks,

I haven't seen it mentioned here (hopefully I wasn't too blind to see):

If I run "su someuser" as root, then it gets stuck. No prompt. I cannot interrupt it with ^C or ^Z either. /var/log/auth.log says:

May 13 08:06:41 pluto su[4193]: Successful su for root by harri
May 13 08:06:41 pluto su[4193]: + pts/3 harri:root
May 13 08:06:41 pluto su[4193]: pam_unix(su:session): session opened for user root by harri(uid=1000)

ps shows:

% ps -ef | grep pts/3

harri     4007  4006  0 07:58 pts/3    00:00:00 bash
root      4193  4007  0 08:06 pts/3    00:00:00 su
root      4194  4193  0 08:06 pts/3    00:00:00 bash
root      4209  4194  0 08:08 pts/3    00:00:00 su
root      4210  4209  0 08:08 pts/3    00:00:00 bash
root      4217  4210  0 08:08 pts/3    00:00:00 stty intr ^C

So obviously 'stty' is to blame here (called from root's .bashrc, as it seems). But for 2.6.24.4 there is no such problem.

Maybe you could spread some light here?

Regards

Harri

#1 lore raw

Hi,

Quoted 6 lines · Harald Dunkel

Hi folks,

I haven't seen it mentioned here (hopefully I wasn't too blind to see):

If I run "su someuser" as root, then it gets stuck. No prompt. I cannot interrupt it with ^C or ^Z either. /var/log/auth.log says:

...

Quoted 5 lines

So obviously 'stty' is to blame here (called from root's .bashrc, as it seems). But for 2.6.24.4 there is no such problem.

Maybe you could spread some light here?

Can you try to run "strace su someuser" instead?

This should produce some screenfulls of information, but will also pinpoint the location of the hang in userspace if it's an error somewhere in the path of a system call. Strace will output to stderr, so use 2>strace.txt to capture it to a file. If you do this as root, you should not have to enter any passwords, so they won't be logged either.

Vegard

Signature · 4 lines
"The animistic metaphor of the bug that maliciously sneaked in while
the programmer was not looking is intellectually dishonest as it
disguises that the error is the programmer's own creation."
	-- E. W. Dijkstra, EWD1036
#2 lore raw
Quoted 3 lines · Vegard Nossum

Can you try to run "strace su someuser" instead?

I tried, but its a Heisenbug: If I run it with strace (with or without -f), then it doesn't get stuck.

Regards

Harri

#3 lore raw
Quoted 7 lines · Harald Dunkel
Quoted 3 lines · Vegard Nossum

Can you try to run "strace su someuser" instead?

I tried, but its a Heisenbug: If I run it with strace (with or without -f), then it doesn't get stuck.

even as root ? Because if you run strace on "su" as a user, you will lose the setuid which may change the conditions.

Willy

#4 lore raw
Quoted 4 lines · Willy Tarreau

even as root ? Because if you run strace on "su" as a user, you will lose the setuid which may change the conditions.

It gets stuck _only_ if I am root. Running su without root permission is no problem.

As written before, If I kick out the "stty intr '^C'" from root's .bashrc, then the second su succeeds. Running the stty command manually in this new shell is no problem.

Can you reproduce this problem? It doesn't need XWindow or any special user. Simply login as root on tty2, and run "su". If it gets stuck, then you could login on tty3 and kill -9 ### the blocking stty.

Regards

Harri

#5 lore raw
Quoted 8 lines · Harald Dunkel
Quoted 4 lines · Willy Tarreau

even as root ? Because if you run strace on "su" as a user, you will lose the setuid which may change the conditions.

It gets stuck _only_ if I am root. Running su without root permission is no problem.

OK. That's quite strange.

Quoted 8 lines

As written before, If I kick out the "stty intr '^C'" from root's .bashrc, then the second su succeeds. Running the stty command manually in this new shell is no problem.

Can you reproduce this problem? It doesn't need XWindow or any special user. Simply login as root on tty2, and run "su". If it gets stuck, then you could login on tty3 and kill -9 ### the blocking stty.

I tried (I'm on 2.6.25.1). But neither "su", "su willy" nor "stty intr ^C" caused such a problem unfortunately.

You may want to ask several people to test in the same environment as yours (same distro, etc...) in order to find whether it's a user-land bug or a kernel bug. Most likely it's a kernel bug or side effect which is triggered by your version of su or stty.

Quoted 3 lines

Regards

Harri

Willy

#6 lore raw
In reply to Harald Dunkel
Quoted 6 lines · Harald Dunkel
Quoted 1 line · Vegard Nossum

Can you try to run "strace su someuser" instead?

I tried, but its a Heisenbug: If I run it with strace (with or without -f), then it doesn't get stuck.

Ah, tough luck. What if you attach an strace to the already stuck process? Like strace -p pid. It's a long shot, but worth a try, I suppose.

You could also try to use sysrq to output some useful information about the stuck process. See Documentation/sysrq.txt for more information. Gotta run, good luck!

Vegard

Signature · 4 lines
"The animistic metaphor of the bug that maliciously sneaked in while
the programmer was not looking is intellectually dishonest as it
disguises that the error is the programmer's own creation."
	-- E. W. Dijkstra, EWD1036
#7 lore raw
Quoted 5 lines · Vegard Nossum

Ah, tough luck. What if you attach an strace to the already stuck process? Like strace -p pid. It's a long shot, but worth a try, I suppose.

As suggested I attached strace to the blocking stty. I got a continuous flow of

:
:
ioctl(0, SNDCTL_TMR_STOP or TCSETSW, {B38400 opost isig icanon echo ...}) = ? ERESTARTSYS (To be restarted) --- SIGTTOU (Stopped (tty output)) @ 0 (0) ---
--- SIGTTOU (Stopped (tty output)) @ 0 (0) ---
:
:

I also moved back to an old version of coreutils from about 4 months ago. Same result.

Regards

Harri

#8 lore raw
Quoted 19 lines · Harald Dunkel
Quoted 5 lines · Vegard Nossum

Ah, tough luck. What if you attach an strace to the already stuck process? Like strace -p pid. It's a long shot, but worth a try, I suppose.

As suggested I attached strace to the blocking stty. I got a continuous flow of

:
:
ioctl(0, SNDCTL_TMR_STOP or TCSETSW, {B38400 opost isig icanon echo ...}) = ? ERESTARTSYS (To be restarted)
--- SIGTTOU (Stopped (tty output)) @ 0 (0) ---
--- SIGTTOU (Stopped (tty output)) @ 0 (0) ---
:
:

Quoted 2 lines

I also moved back to an old version of coreutils from about 4 months ago. Same result.

Looks good. Adding Alan Cox to conversation. Original report:

--------
If I run "su someuser" as root, then it gets stuck. No prompt. I cannot interrupt it with ^C or ^Z either. /var/log/auth.log says:

May 13 08:06:41 pluto su[4193]: Successful su for root by harri
May 13 08:06:41 pluto su[4193]: + pts/3 harri:root
May 13 08:06:41 pluto su[4193]: pam_unix(su:session): session opened

for user root by harri(uid=1000)

ps shows:

% ps -ef | grep pts/3

harri     4007  4006  0 07:58 pts/3    00:00:00 bash
root      4193  4007  0 08:06 pts/3    00:00:00 su
root      4194  4193  0 08:06 pts/3    00:00:00 bash
root      4209  4194  0 08:08 pts/3    00:00:00 su
root      4210  4209  0 08:08 pts/3    00:00:00 bash
root      4217  4210  0 08:08 pts/3    00:00:00 stty intr ^C

So obviously 'stty' is to blame here (called from root's .bashrc, as it seems). But for 2.6.24.4 there is no such problem.
--------

Vegard

Signature · 4 lines
"The animistic metaphor of the bug that maliciously sneaked in while
the programmer was not looking is intellectually dishonest as it
disguises that the error is the programmer's own creation."
	-- E. W. Dijkstra, EWD1036
#9 lore raw
Quoted 2 lines

If I run "su someuser" as root, then it gets stuck. No prompt. I cannot interrupt it with ^C or ^Z either. /var/log/auth.log says:

Is it then killable remotely, can you gdb it ?

#10 lore raw
Quoted 5 lines · Alan Cox
Quoted 2 lines

If I run "su someuser" as root, then it gets stuck. No prompt. I cannot interrupt it with ^C or ^Z either. /var/log/auth.log says:

Is it then killable remotely, can you gdb it ?

Attaching to the running stty failed:

# gdb stty 26465
GNU gdb 6.8-debian
Copyright (C) 2008 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu"...
Attaching to program: /var/tmp/coreutils-6.10/coreutils-6.10/src/stty, process 26465
/tmp/buildd/gdb-6.8/gdb/linux-nat.c:988: internal-error: linux_nat_attach: Assertion `pid == GET_PID (inferior_ptid) && WIFSTOPPED (status) && WSTOPSIG (status) == SIGSTOP' failed.
A problem internal to GDB has been detected,
further debugging may prove unreliable.
:

I can kill it with -9, but INTR, SEGV, BUS, etc. don't interrupt it. And if I run stty inside of the debugger (or on the command line), then it doesn't get stuck.

I have changed stty.c to do some print statements, and changed root's .bashrc to use it for "stty intr ^C": It gets stuck inside tcsetattr():

      printf("here we are\n");
      if (tcsetattr (STDIN_FILENO, TCSADRAIN, &mode))
	error (EXIT_FAILURE, errno, "%s", device_name);
      printf("born to be kings\n");

The previous tcgetattr() to read the old terminal settings shows no problem (AFAICS).

Replacing TCSADRAIN by 0 doesn't help.

Regards

Harri

#11 lore raw
Quoted 4 lines

This GDB was configured as "x86_64-linux-gnu"...
Attaching to program: /var/tmp/coreutils-6.10/coreutils-6.10/src/stty, process 26465
/tmp/buildd/gdb-6.8/gdb/linux-nat.c:988: internal-error: linux_nat_attach: Assertion `pid == GET_PID (inferior_ptid) && WIFSTOPPED (status) && WSTOPSIG (status) == SIGSTOP' failed. A problem internal to GDB has been detected,

So your gdb is broken, your stty is doign stuff nobody else seems to be seeing ? My first thought is to suspect the distro/source as I've still had no other equivalent reports and given the gdb spew.

I'll have a dig further however as the GDB spew itself might actually be a useful clue. Not sure I can do much without a proper trace however (if gdb is borked on your box is strace giving sane reports ?)

Alan

#12 lore raw
Quoted 9 lines · Alan Cox
Quoted 4 lines

This GDB was configured as "x86_64-linux-gnu"...
Attaching to program: /var/tmp/coreutils-6.10/coreutils-6.10/src/stty, process 26465
/tmp/buildd/gdb-6.8/gdb/linux-nat.c:988: internal-error: linux_nat_attach: Assertion `pid == GET_PID (inferior_ptid) && WIFSTOPPED (status) && WSTOPSIG (status) == SIGSTOP' failed. A problem internal to GDB has been detected,

So your gdb is broken, your stty is doign stuff nobody else seems to be seeing ? My first thought is to suspect the distro/source as I've still had no other equivalent reports and given the gdb spew.

I doubt that Debian is to blame here. I get the same with native gdb-6.8 and native coreutils-6.10:

# /usr/local/stow/gdb-6.8/bin/gdb ./stty 11944
GNU gdb 6.8
Copyright (C) 2008 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-unknown-linux-gnu"...
Attaching to program: /var/tmp/coreutils-6.10/coreutils-6.10/src/stty, process 11944
../../gdb/linux-nat.c:988: internal-error: linux_nat_attach: Assertion `pid == GET_PID (inferior_ptid) && WIFSTOPPED (status) && WSTOPSIG (status) == SIGSTOP' failed.
A problem internal to GDB has been detected,
further debugging may prove unreliable.
Quit this debugging session? (y or n) y

../../gdb/linux-nat.c:988: internal-error: linux_nat_attach: Assertion `pid == GET_PID (inferior_ptid) && WIFSTOPPED (status) && WSTOPSIG (status) == SIGSTOP' failed. A problem internal to GDB has been detected,
further debugging may prove unreliable.
Create a core file of GDB? (y or n) n

Quoted 4 lines

I'll have a dig further however as the GDB spew itself might actually be a useful clue. Not sure I can do much without a proper trace however (if gdb is borked on your box is strace giving sane reports ?)

As written earlier in this thread I can attach strace to the running stty. It prints a continuous flow of

:
ioctl(0, SNDCTL_TMR_START or TCSETS, {B38400 opost isig icanon echo ...}) = ? ERESTARTSYS (To be restarted) --- SIGTTOU (Stopped (tty output)) @ 0 (0) ---
--- SIGTTOU (Stopped (tty output)) @ 0 (0) ---
:

Using strace -v I get

:
ioctl(0, SNDCTL_TMR_START or TCSETS, {c_iflags=0x500, c_oflags=0x5, c_cflags=0xbf, c_lflags=0x8a3b, c_line=0, c_cc="\x03\x1c\x7f\x15\x04\x00\x01\x00\x11\x13\x1a\x00\x12\x0f\x17\x16\x00\x00\x00"}) = ? ERESTARTSYS (To be restarted) --- SIGTTOU (Stopped (tty output)) @ 0 (0) ---
--- SIGTTOU (Stopped (tty output)) @ 0 (0) ---
:

Surely I am no specialist for termio. Please mail if I can help. BTW, I am on 2.6.25.4 now.

Regards

Harri

#13 lore raw
Quoted 6 lines · Harald Dunkel
Quoted 5 lines · Alan Cox
Quoted 4 lines

This GDB was configured as "x86_64-linux-gnu"...
Attaching to program: /var/tmp/coreutils-6.10/coreutils-6.10/src/stty, process 26465
/tmp/buildd/gdb-6.8/gdb/linux-nat.c:988: internal-error: linux_nat_attach: Assertion `pid == GET_PID (inferior_ptid) && WIFSTOPPED (status) && WSTOPSIG (status) == SIGSTOP' failed. A problem internal to GDB has been detected,

Quoted 7 lines
Quoted 4 lines

So your gdb is broken, your stty is doign stuff nobody else seems to be seeing ? My first thought is to suspect the distro/source as I've still had no other equivalent reports and given the gdb spew.

I doubt that Debian is to blame here. I get the same with native gdb-6.8 and native coreutils-6.10:

try killing syslog and then see if you continue to have the problem. It's possible that what's happening is syslog is getting stuck and su is sitting waiting for syslog to process the log entry.

David Lang

#14 lore raw
Quoted 3 lines · [email protected] wrote:

try killing syslog and then see if you continue to have the problem. It's possible that what's happening is syslog is getting stuck and su is sitting waiting for syslog to process the log entry.

No improvement: I killed syslogd, and yet stty gets stuck.

But surely it was a smart idea.

Regards

Harri

#15 lore raw
Quoted 10 lines · Harald Dunkel
Quoted 5 lines · [email protected] wrote:

try killing syslog and then see if you continue to have the problem. It's possible that what's happening is syslog is getting stuck and su is sitting waiting for syslog to process the log entry.

No improvement: I killed syslogd, and yet stty gets stuck.

But surely it was a smart idea.

have you checked in /proc/$(pidof stty)/fd/ to see if stty is bound to any particular file descriptor ? Maybe you'll find the source of the blocking operation here. You can find a unix socket attached to another process, it may be a tty, a pipe, or even a device (eg:
/dev/random when there is no entropy left). Same for "su" BTW.

Regards,
willy