Skip to contentSkip to conversation

3.0.18 tcsetattr on fd 0 when detached freezes system (RCU timeouts) (Centos 6.1 x86_64)

DISCUSSION Professor Berkley Shands 2 participants 4 messages Jan 28, 2012 – Feb 2, 2012 1 message not in the archive

lore
Share & tools
lore
#0 lore raw

typedef struct
{
struct termios term;
} XKEY_DATA;

typedef XKEY_DATA *xkeyhandle;

static inline xkeyhandle xkeystart()
{

   // Turn off echo.
   struct termios temp;
   err = tcgetattr(0, &temp);
   if (err)
   {
      perror("tcgetattr failure");
   }

   XKEY_DATA *handle = new XKEY_DATA;
   handle->term = temp;

   temp.c_lflag &= ~ECHO;
   temp.c_lflag &= ~ICANON;

err = tcsetattr(0, TCSANOW, &temp); // this line causes the
kernel to get very sick
if (err)

   {
      perror("tcsetattr failure");

}

return handle;

}

The above code, called from main() will produce an error from tcsh:

/home/bshands> ./a.out > /dev/null &
[1] 3635
/home/bshands>
/home/bshands>
[1] + Suspended (tty output) ./a.out > /dev/null
/home/bshands>

this does not appear on the redhat kernel, nor 2.6.32.43, but appeared infrequently in 3.0.9. in 3.0.18, doing this in the background does *EVIL* things.

ssh system "./a.out > /dev/null &" &

Now when the code reaches the tcsetattr() the system quits scheduling tasks. top shows 100%sy on 4/12 cores, kernel threads blocked, stalled tasks count increasing.

Jan 27 18:58:50 system kernel: [ 1269.891610] kjournald       S ffff880321270a30     0   416      2 0x00000000
Jan 27 18:58:50 system kernel: [ 1269.891684]  ffff880321f5de50 0000000000000046 ffff880321270680 0000000000000000
Jan 27 18:58:50 system kernel: [ 1269.891816]  ffff880321270680 0000000000012ac0 ffff880321f5dfd8 ffff880321f5c010
Jan 27 18:58:50 system kernel: [ 1269.891948]  ffff880321f5dfd8 0000000000012ac0 ffff880327a61260 ffff880321270680
Jan 27 18:58:50 system kernel: [ 1269.892083] Call Trace:
Jan 27 18:58:50 system kernel: [ 1269.892144]  [<ffffffff824c126f>] schedule+0x3f/0x60
Jan 27 18:58:50 system kernel: [ 1269.892226]  [<ffffffffa0030f9d>] kjournald+0x20d/0x230 [jbd]
Jan 27 18:58:50 system kernel: [ 1269.892294]  [<ffffffff82082230>] ? wake_up_bit+0x40/0x40
Jan 27 18:58:50 system kernel: [ 1269.892362]  [<ffffffffa0030d90>] ? commit_timeout+0x10/0x10 [jbd]
Jan 27 18:58:50 system kernel: [ 1269.892431]  [<ffffffff82081bb6>] kthread+0x96/0xa0
Jan 27 18:58:50 system kernel: [ 1269.892497]  [<ffffffff824cbe04>] kernel_thread_helper+0x4/0x10
Jan 27 18:58:50 system kernel: [ 1269.892565]  [<ffffffff82081b20>] ? kthread_worker_fn+0x1a0/0x1a0
Jan 27 18:58:50 system kernel: [ 1269.892633]  [<ffffffff824cbe00>] ? gs_change+0x13/0x13

After a little while, the system locks up. rcu timeout errors may appear in dmesg. dumps are available.

Note that this little bugger is a user mode crash, once started, you are done. it appears that later use of termios is still sufficient to trigger the lockup. they key is being detached.

Berkley

#1 lore raw
Quoted 44 lines · Professor Berkley Shands

typedef struct

{
   struct termios term;
} XKEY_DATA;

typedef XKEY_DATA *xkeyhandle;

static inline xkeyhandle xkeystart()
{

   // Turn off echo.
   struct termios temp;
   err = tcgetattr(0, &temp);
   if (err)
   {
      perror("tcgetattr failure");
   }

   XKEY_DATA *handle = new XKEY_DATA;
   handle->term = temp;

   temp.c_lflag &= ~ECHO;
   temp.c_lflag &= ~ICANON;

   err = tcsetattr(0, TCSANOW, &temp);        // this line causes the kernel to get very sick
   if (err)
   {
      perror("tcsetattr failure");
   }
   return handle;

}

The above code, called from main() will produce an error from tcsh:

/home/bshands> ./a.out > /dev/null &
[1] 3635
/home/bshands>
/home/bshands>
[1] + Suspended (tty output) ./a.out > /dev/null
/home/bshands>

Quoted 7 lines

this does not appear on the redhat kernel, nor 2.6.32.43, but appeared infrequently in 3.0.9. in 3.0.18, doing this in the background does *EVIL* things.

ssh system "./a.out > /dev/null &" &

Now when the code reaches the tcsetattr() the system quits scheduling tasks. top shows 100%sy on 4/12 cores, kernel threads blocked, stalled tasks count increasing.

I used the following code:

=======================================
#include <termios.h>
#include <stdio.h>

int main() {
struct termios temp;

   if (tcgetattr(0, &temp) != 0)
      perror("tcgetattr failure");

   temp.c_lflag &= ~ECHO;
   temp.c_lflag &= ~ICANON;

if (tcsetattr(0, TCSANOW, &temp) != 0) // this line causes the kernel to get very sick

      perror("tcsetattr failure");

return 0;
}
=======================================

But can't reproduce what you're observing. It prints

tcgetattr failure: Inappropriate ioctl for device tcsetattr failure: Inappropriate ioctl for device

and does not do any evil things. I tried it on 3.0.18 on x86 on 32bits and 64bits.

What I'm doing wrong?

Thanks,

/mjt

A message that is not in the archive (sent off-list or not yet received): [email protected]

#3 lore raw
Quoted 3 lines · Professor Berkley Shands

Very strange. Now boxes with NO OFED, no Intel-10 Gige and no special drivers are locking up. If your kernel is Red Hat compatible, could you please send me a copy of the .config so I can try to isolate this more?

Why are you writing to me personally? I just tried your code and can't find the problem you see, and I replied to the list. Cc'ing to the list now.

I don't understand what is "redhat compatible". I use kernel from kernel.org, currently at version 3.0.18.

Did you try my small "reproducer" - does it lock your machines too? I provided complete code which is compilable and runnable, unlike your version which lacked some context.

Thanks,

/mjt

#4 lore raw

I built my .config from the redhat .config provided in 2.6.32-131 using make oldconfig. that failed miserably. I then used one based on 2.6.39.4, which actually booted, but I get these lockup errors, RCU timeouts, ...

The system died right away on the tcsetattr(), (which also did not return any error). And my simple test case crashed all the time. Looked rather suspicous...
Now after a week, *ALL* my 3.0.18 boxes lock up (other than sitting IDLE, any load eventually causes the system to stop scheduling). That is 32 core 6282's, 3.46GHz Nehalems, 2.3 GHz 2374's... I have to assume the tcsetattr() is an artifact at this point.
Without building all the kernels in between 2.6.32.55 and 3.0.18, I needed a starting point for the .config that works. Usually it is something unnoticed that needed to be updated that make oldconfig didn't point out. Things do change that I can't keep current on. :-)

So it appears that it has to be my configuration. Hence the request for a .config I can compare against to see what is wrong / misconfigured / not configured etc.

Berkley