Skip to contentSkip to conversation

Xterm Hangs - Possible scheduler defect?

DISCUSSION Chad N. Tindel 6 participants 13 messages Feb 23, 2005 – Feb 25, 2005

lore
Share & tools
lore
#0 lore raw

Hello-

We have hit a defect where an exiting xterm process will hang. This is running on a 2-cpu IA-64 box. We have a multithreaded application, where one thread is SCHED_FIFO and is running with priority 98, and the other thread is just a normal SCHED_OTHER thread. The SCHED_FIFO thread is in a CPU bound tight loop, but I wouldn't expect that to cause since there are 2 CPUs.

However, it does seem to cause some problems. For example, if you ssh into the system and run an Xterm using X11 forwarding, when you type "exit" in the xterm window, the window hangs and doesn't close. Killing the CPU-bound app causes the window to exit immediately. The sysrq output shows the following:

xterm D a0000001000bef60 0 2905 2876 (NOTLB)

Call Trace:
 [<a0000001004ac480>] schedule+0xca0/0x1300
                                sp=e000000012257d20 bsp=e000000012251080
 [<a0000001000bef60>] flush_cpu_workqueue+0x1a0/0x4a0
                                sp=e000000012257d30 bsp=e000000012251020
 [<a0000001000bf360>] flush_workqueue+0x100/0x160
                                sp=e000000012257d90 bsp=e000000012250fe8
 [<a0000001000bfd60>] flush_scheduled_work+0x20/0x40
                                sp=e000000012257d90 bsp=e000000012250fd0
 [<a0000001002e2060>] release_dev+0x8e0/0x1100
                                sp=e000000012257d90 bsp=e000000012250f20
 [<a0000001002e3350>] tty_release+0x30/0x60
                                sp=e000000012257e30 bsp=e000000012250ef8
 [<a00000010012d430>] __fput+0x330/0x340
                                sp=e000000012257e30 bsp=e000000012250ea8
 [<a00000010012d0e0>] fput+0x40/0x60
                                sp=e000000012257e30 bsp=e000000012250e88
 [<a00000010012a1b0>] filp_close+0xd0/0x160
                                sp=e000000012257e30 bsp=e000000012250e58
 [<a00000010012a380>] sys_close+0x140/0x1a0
                                sp=e000000012257e30 bsp=e000000012250dd8
 [<a00000010000aba0>] ia64_ret_from_syscall+0x0/0x20
                                sp=e000000012257e30 bsp=e000000012250dd8

So it would appear that xterm is hung in close() trying to shutdown a tty. The comment says that is calling flush_scheduled_work() to
"Wait for ->hangup_work and ->flip.work handlers to terminate". Perhaps there is some locking issue that is causing these to not run and complete?

I'm a bit out of my space here... does anybody have any ideas? I've tried this on both 2.6.8 and 2.6.10 with the same problem resulting.

Please make sure to CC me in any responses.

Regards,

Chad

#1 lore raw
Quoted 39 lines · Chad N. Tindel

We have hit a defect where an exiting xterm process will hang. This is running on a 2-cpu IA-64 box. We have a multithreaded application, where one thread is SCHED_FIFO and is running with priority 98, and the other thread is just a normal SCHED_OTHER thread. The SCHED_FIFO thread is in a CPU bound tight loop, but I wouldn't expect that to cause since there are 2 CPUs.

However, it does seem to cause some problems. For example, if you ssh into the system and run an Xterm using X11 forwarding, when you type "exit" in the xterm window, the window hangs and doesn't close. Killing the CPU-bound app causes the window to exit immediately. The sysrq output shows the following:

xterm D a0000001000bef60 0 2905 2876 (NOTLB)

 Call Trace:
  [<a0000001004ac480>] schedule+0xca0/0x1300
                                 sp=e000000012257d20 bsp=e000000012251080
  [<a0000001000bef60>] flush_cpu_workqueue+0x1a0/0x4a0
                                 sp=e000000012257d30 bsp=e000000012251020
  [<a0000001000bf360>] flush_workqueue+0x100/0x160
                                 sp=e000000012257d90 bsp=e000000012250fe8
  [<a0000001000bfd60>] flush_scheduled_work+0x20/0x40
                                 sp=e000000012257d90 bsp=e000000012250fd0
  [<a0000001002e2060>] release_dev+0x8e0/0x1100
                                 sp=e000000012257d90 bsp=e000000012250f20
  [<a0000001002e3350>] tty_release+0x30/0x60
                                 sp=e000000012257e30 bsp=e000000012250ef8
  [<a00000010012d430>] __fput+0x330/0x340
                                 sp=e000000012257e30 bsp=e000000012250ea8
  [<a00000010012d0e0>] fput+0x40/0x60
                                 sp=e000000012257e30 bsp=e000000012250e88
  [<a00000010012a1b0>] filp_close+0xd0/0x160
                                 sp=e000000012257e30 bsp=e000000012250e58
  [<a00000010012a380>] sys_close+0x140/0x1a0
                                 sp=e000000012257e30 bsp=e000000012250dd8
  [<a00000010000aba0>] ia64_ret_from_syscall+0x0/0x20
                                 sp=e000000012257e30 bsp=e000000012250dd8
Quoted 4 lines

So it would appear that xterm is hung in close() trying to shutdown a tty. The comment says that is calling flush_scheduled_work() to
"Wait for ->hangup_work and ->flip.work handlers to terminate". Perhaps there is some locking issue that is causing these to not run and complete?

xterm' is waiting for the other CPU to schedule a kernel thread (which is bound to that CPU). Once that kernel thread has done a little bit of work, xterm' can terminate.

But kernel threads don't run with realtime policy, so your userspace app has permanently starved that kernel thread.

It's potentially quite a problem, really. For example it could prevent various tty operations from completing, it will prevent kjournald from ever writing back anything (on uniprocessor, etc). I've been waiting for someone to complain ;)

But the other side of the coin is that a SCHED_FIFO userspace task presumably has extreme latency requirements, so it doesn't *want* to be preempted by some routine kernel operation. People would get irritated if we were to do that.

So what to do?

#2 lore raw
Quoted 12 lines

xterm' is waiting for the other CPU to schedule a kernel thread (which is bound to that CPU). Once that kernel thread has done a little bit of work, xterm' can terminate.

But kernel threads don't run with realtime policy, so your userspace app has permanently starved that kernel thread.

It's potentially quite a problem, really. For example it could prevent various tty operations from completing, it will prevent kjournald from ever writing back anything (on uniprocessor, etc). I've been waiting for someone to complain ;)

Quoted 6 lines

But the other side of the coin is that a SCHED_FIFO userspace task presumably has extreme latency requirements, so it doesn't *want* to be preempted by some routine kernel operation. People would get irritated if we were to do that.

So what to do?

It shouldn't need to preempt the kernel operation. Why is the design such that the necessary kernel thread can't run on the other CPU?

Chad

#4 lore raw
In reply to Andrew Morton
Quoted 4 lines

But the other side of the coin is that a SCHED_FIFO userspace task presumably has extreme latency requirements, so it doesn't *want* to be preempted by some routine kernel operation. People would get irritated if we were to do that.

Just to follow up a bit. People writing apps that run at SCHED_FIFO know that they aren't getting hard real-time, and they are OK with that. If they wanted something more they'd run on RTLinux. Why would it be wrong to preempt the SCHED_FIFO process in the case, assuming that it is too hard to fix a broken design that doesn't allow the necessary kernel threads to run on any CPU?

Chad

#3 lore raw
In reply to Chad N. Tindel
Quoted 18 lines · Chad N. Tindel
Quoted 18 lines

xterm' is waiting for the other CPU to schedule a kernel thread (which is bound to that CPU). Once that kernel thread has done a little bit of work, xterm' can terminate.

But kernel threads don't run with realtime policy, so your userspace app has permanently starved that kernel thread.

It's potentially quite a problem, really. For example it could prevent various tty operations from completing, it will prevent kjournald from ever writing back anything (on uniprocessor, etc). I've been waiting for someone to complain ;)

But the other side of the coin is that a SCHED_FIFO userspace task presumably has extreme latency requirements, so it doesn't *want* to be preempted by some routine kernel operation. People would get irritated if we were to do that.

Quoted 5 lines
Quoted 1 line

So what to do?

It shouldn't need to preempt the kernel operation. Why is the design such that the necessary kernel thread can't run on the other CPU?

This particular kernel function is implemented via a kernel thread per CPU, with each thread bound to each CPU. The xterm-does-exit cleanup code is waiting for the thread which is bound to the busy CPU to do something.

No other CPU can, or is allowed, to do that thread's work. If it were to do so, the implicit locking which we get from the per-cpuness would be violated.

I don't know if any clients of the workqueue code rely upon the pinned-to-cpu feature.

#5 lore raw
In reply to Chad N. Tindel
Quoted 7 lines · Chad N. Tindel
Quoted 6 lines

But the other side of the coin is that a SCHED_FIFO userspace task presumably has extreme latency requirements, so it doesn't *want* to be preempted by some routine kernel operation. People would get irritated if we were to do that.

Quoted 7 lines

Just to follow up a bit. People writing apps that run at SCHED_FIFO know that they aren't getting hard real-time, and they are OK with that. If they wanted something more they'd run on RTLinux. Why would it be wrong to preempt the SCHED_FIFO process in the case, assuming that it is too hard to fix a broken design that doesn't allow the necessary kernel threads to run on any CPU?

Why would anyone write a thread than uses exactly 100% of one cpu?
It seems wrong to me. It is unsafe if they really need that much
processing power, what if an interrupt happens? Then they get slightly less.
If they got a 10% faster cpu, would this thread suddenly drop to 90%
usage (and no problems with kernel threads)? If it stays at 100% then that suggests they are using some sort of busy waiting which is bad programming. Get a better hw
device that will provide an interrupt at the right time, and write a driver for
that. Or find some spot in the code where a small delay in acceptable,
and set a short timer and sleep on it. It doesn't take much to get this
kernel thread going.

Helge Hafting

#6 lore raw
Quoted 4 lines

Why would anyone write a thread than uses exactly 100% of one cpu? It seems wrong to me. It is unsafe if they really need that much processing power, what if an interrupt happens? Then they get slightly less. If they got a 10% faster cpu, would this thread suddenly drop to 90%

Quoted 8 lines

usage (and no problems with kernel threads)?
If it stays at 100% then that suggests they are using some sort of busy waiting which is bad programming. Get a better hw device that will provide an interrupt at the right time, and write a driver for
that. Or find some spot in the code where a small delay in acceptable, and set a short timer and sleep on it. It doesn't take much to get this kernel thread going.

I would make the following assertion for any kernel:

No single userspace thread of execution running on an SMP system should be able to hose a box by going CPU-bound, bug in the software or no bug. Any kernel should be able to handle this case and shift general work over to other processors.

While I can't speak for all commercial Unixes, I know that HP-UX handles this case just fine. I'd be extremely surprised if Solaris and AIX didn't handle it fine too. What I can't understand is why you want to cop-out and say "Oh well this is just a bug in the application... the programmer shouldn't shoot himself in the foot." If that were the attitude that kernel programmers had, why have the kernel send SIGSEGV when applications reference invalid memory? Why not just let them corrupt the memory of other apps and possibly bring the whole system down?

It is the kernel's job to protect itself and userspace applications from runaway applications whenever possible. While this might not be possible for this case on a UP box, it certainly is for an SMP box.

Chad

#7 lore raw
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

"Chad" == Chad N Tindel <[email protected]> writes:

Chad> I would make the following assertion for any kernel:

Chad> No single userspace thread of execution running on an SMP system Chad> should be able to hose a box by going CPU-bound, bug in the Chad> software or no bug. Any kernel should be able to handle this Chad> case and shift general work over to other processors.

In many Unices, crucial kernel threads run at realtime priority with a static priority higher than is accessible to user code.

That being said, however, you've got to be a privileged user to set real time very high priority on a thread, and if you do, you'd better know what you're doing. Any SCHED_FIFO thread should run for a time, then sleep for a time, or it *will* DOS everything else on the processor.

--
Dr Peter Chubb http://www.gelato.unsw.edu.au peterc AT gelato.unsw.edu.au The technical we do immediately, the political takes *forever*

#8 lore raw
Quoted 2 lines

In many Unices, crucial kernel threads run at realtime priority with a static priority higher than is accessible to user code.

Yep.

Quoted 5 lines

That being said, however, you've got to be a privileged user to set real time very high priority on a thread, and if you do, you'd better know what you're doing. Any SCHED_FIFO thread should run for a time, then sleep for a time, or it *will* DOS everything else on the processor.

This is only true if you're not doing what you said in your first paragraph, i.e. running crucial kernel threads higher than any user thread.

Chad

#9 lore raw
In reply to Chad N. Tindel
Quoted 5 lines · Chad N. Tindel

I would make the following assertion for any kernel:

No single userspace thread of execution running on an SMP system should be able to hose a box by going CPU-bound, bug in the software or no bug.

But if we were to enforce that policy, realtime policy would become less useful. You havn't even acknowledged that such a tradeoff exists, let alone demonstrated that we're on the wrong side of it.

Here's a quicky which will convert all your kernel threads to SCHED_RR, priority 99. Please test.

#!/bin/sh

PIDS=$(ps axo pid,command | grep ' \[.*\]$' | sed -e 's/ \[.*\]$//')

for i in $PIDS
do

	chrt -r 99 -9 $i

done

#10 lore raw
Quoted 8 lines · Andrew Morton

#!/bin/sh

PIDS=$(ps axo pid,command | grep ' \[.*\]$' | sed -e 's/ \[.*\]$//')

for i in $PIDS
do

	chrt -r 99 -9 $i

done

For the unaware, "chrt" is part of the schedutils package. (I didn't know about it till just now...figured I'd save others the trouble of searching.)

Chris

#11 lore raw
Quoted 4 lines · Chris Friesen
Quoted 1 line · Andrew Morton
	chrt -r 99 -9 $i

Make that

	chrt -r 99 -p $i
#12 lore raw
In reply to Andrew Morton
Quoted 2 lines · Andrew Morton

Here's a quicky which will convert all your kernel threads to SCHED_RR, priority 99. Please test.

We have a bunch of workstations here where we run a similar thing during boot, as well as starting a SCHED_RR @ 99 sulogin-type process on tty12. It makes blasting the occasional annoying fork-bomb or CPU-chewing-crashed-X a lot nicer.

Cheers,
Kyle Moffett

-----BEGIN GEEK CODE BLOCK-----
Version: 3.12
GCM/CS/IT/U d- s++: a18 C++++>$ UB/L/X/*++++(+)>$ P+++(++++)>$ L++++(+++) E W++(+) N+++(++) o? K? w--- O? M++ V? PS+() PE+(-) Y+ PGP+++ t+(+++) 5 X R? tv-(--) b++++(++) DI+ D+ G e->++++$ h!*()>++$ r !y?(-) ------END GEEK CODE BLOCK------