Skip to contentSkip to conversation

Xterm Hangs - Possible scheduler defect?

DISCUSSION Chad N. Tindel 8 participants 20 messages Feb 24, 2005 – Feb 26, 2005 2 messages not in the archive

lore
Share & tools
lore
#2 lore raw
In reply to a message not in the archive
Quoted 9 lines
Quoted 3 lines

Hmmm... Are you suggesting it is OK for a kernel to get nearly completely hosed and for not fully utilize all the processors in the system because of one SCHED_FIFO thread?

Sure. You specifically directed the scheduler to run your thread at a higher priority than anything else. The way I see it, you used root's perogative to shoot himself in the foot. You could also have used root's perogative to don steel toed shoes(set important kernel threads to a higher priority) before pulling the trigger.

No, I specifically directed the scheduler to run my thread at a higher priority than any other userspace application. The fact that I wrote it in userspace and not in kernel space implies that I am OK with the kernel stopping me sometimes when _it_ has work to do. If I wanted something higher priority than the kernel I would have written something in kernel space instead.

Quoted 5 lines

SCHED_FIFO thread are supposed to preempt

Quoted 1 line

all other userspace threads... not the kernel itself.

Not so. The scheduler makes do distinction between user and kernel threads of execution.

That is SOOOO broken it isn't even funny.

Quoted 5 lines

If you think that's broken, you'll _love_ Ingo's IRQ threads. While testing one of his recent (slightly buggy)unpriveleged-user-does-RT patches in an IRQ threadified kernel, I ran a user SCHED_FIFO task at higher than the IRQ0 thread... if my box had been an embeded washing machine controller instead of a desktop box, it'd have been a classic case of "No tickie no washie" :))

Yeah, thats broken too.

Perhaps I don't understand this philosophy you have where the kernel isn't more important than everything else. It seems to me like there needs to be a rigid hierarchy for scheduling, lest you get into deadlock problems:

1. Kernel preempts all. There may be some hierarchy of kernel priorities too, but it isn't important here.

2.  SCHED_FIFO processes preempt all userspace applications.
3.  SCHED_RR.
4.  SCHED_OTHER.

Under no circumstances should any single CPU-bound userspace thread completely hose a 64-way SMP box.

Can somebody educate me on why it is correct to do it any other way?

Chad

#3 lore raw
Quoted 9 lines · Chad N. Tindel

1. Kernel preempts all. There may be some hierarchy of kernel priorities too, but it isn't important here.

2.  SCHED_FIFO processes preempt all userspace applications.
3.  SCHED_RR.
4.  SCHED_OTHER.

Under no circumstances should any single CPU-bound userspace thread completely hose a 64-way SMP box.

Can somebody educate me on why it is correct to do it any other way?

Low-latency userspace apps. The audio guys, for instance, are trying to get latencies down to the 100us range.

If random kernel threads can preempt userspace at any time, they wreak havoc with latency as seen by userspace.

Chris

#4 lore raw
Quoted 5 lines

Low-latency userspace apps. The audio guys, for instance, are trying to get latencies down to the 100us range.

If random kernel threads can preempt userspace at any time, they wreak havoc with latency as seen by userspace.

Come now. There is no such thing as a random kernel thread. Any General Purpose kernel needs the ability to do work that keeps the entire system from grinding to a halt.

Chad

#5 lore raw
Quoted 7 lines · Chad N. Tindel
Quoted 3 lines

Low-latency userspace apps. The audio guys, for instance, are trying to get latencies down to the 100us range.

If random kernel threads can preempt userspace at any time, they wreak havoc with latency as seen by userspace.

Come now. There is no such thing as a random kernel thread. Any General
Purpose kernel needs the ability to do work that keeps the entire system from grinding to a halt.

FYI most kernel threads do background work, that doesn't have hard real-time constraints. Why should my audio recording session get interrupted (read: "sent to the trashcan") just because the swap daemon decided that it was a good time to write some pages out? Couldn't it have waited just a few more milliseconds?

You don't seem to realize that you have just arrived to this mailing list and missed years of discussions on kernel architecture.

If you keep a learning attitude, there is a chance for this discussion to go on. However, if you keep the "Come now, don't bullshit me, this is a broken architecture and you're just trying to cover up" attitude, you're just going to get discarded as a troll.

I personally like the linux way: "root has the ability to shoot himself in the foot if he wants to". This is my computer, damn it, I am the one who tells it what to do.

This is much, much better than the "users are stupid, we must protect them from themselves" kind of way that other OS'es use.

Just my 0.02 euros,

Signature · 4 lines
Paulo Marques - www.grupopie.com

All that is necessary for the triumph of evil is that good men do nothing.
Edmund Burke (1729 - 1797)
#6 lore raw
Quoted 4 lines

If you keep a learning attitude, there is a chance for this discussion to go on. However, if you keep the "Come now, don't bullshit me, this is a broken architecture and you're just trying to cover up" attitude, you're just going to get discarded as a troll.

I'm not trying to troll here; I suppose I'm just coming from a different background. I'll try to adjust my tone.

Quoted 3 lines

I personally like the linux way: "root has the ability to shoot himself in the foot if he wants to". This is my computer, damn it, I am the one who tells it what to do.

I'm all for allowing people to shoot themselves in the foot. That doesn't mean that it is OK for a single userspace thread to mess up a 64-way box.

Quoted 2 lines

This is much, much better than the "users are stupid, we must protect them from themselves" kind of way that other OS'es use.

Isn't this what the kernel attempts to do in many other places? What else could possibly be the point of sending SIGSEGV and causing applications to dump core when they make a mistake referencing memory? Isn't it the kernel's job to protect one application from another?

I see what you're saying about the swap daemon. How about if I make my statement less black and white. Some kernel threads should always have priority over userspace.

I believe the mindset required for a home system that is doing audio recordings is different than the one for enterprise-level systems. How do we unify the two?

Chad

#7 lore raw
Quoted 4 lines · Chad N. Tindel
Quoted 1 line

I personally like the linux way: "root has the ability to shoot himself in the foot if he wants to". This is my computer, damn it, I am the one who tells it what to do.

I'm all for allowing people to shoot themselves in the foot. That doesn't mean that it is OK for a single userspace thread to mess up a 64-way box.

If root has explicitly stated that the thread in question is the highest priority thing on the entire machine, why would it not be okay. The fact that root made a mistake is the issue here, not that the system didn't protect itself.

Quoted 6 lines
Quoted 1 line

This is much, much better than the "users are stupid, we must protect them from themselves" kind of way that other OS'es use.

Isn't this what the kernel attempts to do in many other places? What else could possibly be the point of sending SIGSEGV and causing applications to dump core when they make a mistake referencing memory? Isn't it the kernel's job to protect one application from another?

Yes. But at the same time, if root wants to do something, the kernel should let them. There are many, many ways that root could trash the system. "cat /dev/urandom > /dev/kcore" would do quite nicely.

Quoted 2 lines

I see what you're saying about the swap daemon. How about if I make my statement less black and white. Some kernel threads should always have priority over userspace.

Not necessarily. The latest latency-reduction patches allow root to specify exactly what the priorities should be.

Quoted 3 lines

I believe the mindset required for a home system that is doing audio recordings is different than the one for enterprise-level systems. How do we unify the two?

There are professionals who use linux for audio as well, it's not just home systems. That said, you unify them with reasonable defaults, and the ability for root to override them.

Chris

#16 lore raw
In reply to Chad N. Tindel
Quoted 7 lines
Quoted 2 lines

This is much, much better than the "users are stupid, we must protect them from themselves" kind of way that other OS'es use.

Isn't this what the kernel attempts to do in many other places? What else could possibly be the point of sending SIGSEGV and causing applications to dump core when they make a mistake referencing memory? Isn't it the kernel's job to protect one application from another?

A related example: Typically, when a program (even when running as root) attempts to access I/O ports directly, it gets killed as you describe. However, the X server, running as root, can use ioperm or iopl to request permission to access the video card's I/O ports directly. When it gets that permission, it can do that and no longer gets killed. It also means the X server is capable of bringing the entire system via errant I/O port accesses if it wishes (or if it misbehaves).

The general philosophy is to protect one application from another, unless an application has been specifically granted sufficient power to wreck the system.

I don't remember off the top of my head whether SCHED_FIFO tasks are supposed to be able to take SMP systems down, if the # of SCHED_FIFO tasks is less than the # of CPU's. I imagine someone has thought about this in the past and answered the question one way or another, but I don't happen to know the answer.

-Barry K. Nathan <[email protected]>

#8 lore raw
In reply to Chris Friesen
Quoted 7 lines
Quoted 2 lines

I'm all for allowing people to shoot themselves in the foot. That doesn't mean that it is OK for a single userspace thread to mess up a 64-way box.

If root has explicitly stated that the thread in question is the highest priority thing on the entire machine, why would it not be okay. The fact that root made a mistake is the issue here, not that the system didn't protect itself.

Yeah, I realized when I left for lunch that this statement wasn't as clear as I would like it to be.

I think what we have are the need for two levels of applications:

1. That which wishes to be the highest priority userspace application, and wishes to preempt all other userspace applications. Such an application is OK being preempted by the kernel when the kernel needs to do work. IMHO, this should be the default behavior for any SCHED_FIFO application. If one of these has a bug and goes CPU-bound, the worst it can do is prevent other apps from ever using the CPU it is on.

2. Applications which actually want to be the highest priority thing on the system, including being higher than the kernel. These applications are OK with the fact that they may cause system hangs and deadlocks, and are careful not to shoot themselves in the foot.

Quoted 3 lines

There are professionals who use linux for audio as well, it's not just home systems. That said, you unify them with reasonable defaults, and the ability for root to override them.

OK. Would you say it would be a reasonable default to have SCHED_FIFO userspace threads running at a lower priority than essential kernel threads (say, the load balancer and the events thread), and give root the ability to explicitly have userspace threads preempt the kernel?

Chad

#9 lore raw
Quoted 3 lines · Chad N. Tindel

OK. Would you say it would be a reasonable default to have SCHED_FIFO userspace
threads running at a lower priority than essential kernel threads (say, the
load balancer and the events thread), and give root the ability to explicitly have userspace threads preempt the kernel?

The current scheduler has a 1-100 priority range for soft realtime tasks. To insert a task into a realtime class, you need to have root privileges.

As long as you make sure that kernel threads get set to higher priorities than your user threads, then you get the above behaviour. Ultimately, however, the administrator is responsable for ensuring that everything is running with sane priority levels.

Chris

#10 lore raw
In reply to Chad N. Tindel
Quoted 8 lines · Chad N. Tindel

I think what we have are the need for two levels of applications:

1. That which wishes to be the highest priority userspace application, and wishes to preempt all other userspace applications. Such an application is OK being preempted by the kernel when the kernel needs to do work. IMHO, this should be the default behavior for any SCHED_FIFO application. If one of these has a bug and goes CPU-bound, the worst it can do is prevent other apps from ever using the CPU it is on.

That is basically, what you do with SCHED_RR.
(Be preempted after maximum quantum, even if having work to do)

Quoted 4 lines

2. Applications which actually want to be the highest priority thing on the system, including being higher than the kernel. These applications are OK with the fact that they may cause system hangs and deadlocks, and are careful not to shoot themselves in the foot.

This is SCHED_FIFO.
(Strict priority scheduling, allowed to starve anything below)

So just try to use the right scheduler for your application right now, ok?

If your system is busy with top priority task, why should the kernel disturb it?

Things will stop anyway, if your high priority task is needing a resource, which is blocked. Than it becomes unrunnable and other tasks have chances to continue. Kernel threads are likely to execute then, because they are likely runnable then. Your task could even migrate, if a lot of kernel tasks
are waiting in one CPU and your task is NOT bound to a specific CPU.

So the system is not brought down, but just busy in a infortunate way. Stupid applications can starve other applications for a while, but not forever, because the kernel is still running and deciding.

Regards

Ingo Oeser

#21 lore raw
In reply to Chad N. Tindel
Quoted 10 lines · Chad N. Tindel
Quoted 9 lines
Quoted 3 lines

Hmmm... Are you suggesting it is OK for a kernel to get nearly completely hosed and for not fully utilize all the processors in the system because of one SCHED_FIFO thread?

Sure. You specifically directed the scheduler to run your thread at a higher priority than anything else. The way I see it, you used root's perogative to shoot himself in the foot. You could also have used root's perogative to don steel toed shoes(set important kernel threads to a higher priority) before pulling the trigger.

Quoted 6 lines

No, I specifically directed the scheduler to run my thread at a higher priority than any other userspace application. The fact that I wrote it in userspace and not in kernel space implies that I am OK with the kernel stopping me sometimes when _it_ has work to do. If I wanted something higher priority than the kernel I would have written something in kernel space instead.

Nope. You may have _thought_ you told it that, but the reality is as I described it.

Quoted 7 lines
Quoted 5 lines

SCHED_FIFO thread are supposed to preempt

Quoted 1 line

all other userspace threads... not the kernel itself.

Not so. The scheduler makes do distinction between user and kernel threads of execution.

That is SOOOO broken it isn't even funny.

I heartily disagree. I call it flexible/powerful.

Quoted 3 lines
Quoted 1 line

If you think that's broken, you'll _love_ Ingo's IRQ threads...

Yeah, thats broken too.

(You're not noticing the added power it gives you.)

Quoted 3 lines

Perhaps I don't understand this philosophy you have where the kernel isn't more important than everything else. It seems to me like there needs to be a rigid hierarchy for scheduling, lest you get into deadlock problems:

Some kernel thread flushing buffers should be more important than my userland trigger-pacemaker thread?

Quoted 2 lines

Under no circumstances should any single CPU-bound userspace thread completely hose a 64-way SMP box.

I can certainly agree that any service which is required across processor borders wants to be very high priority indeed, and I can further agree that this crossing of borders would not exist in a perfect world.

-Mike

#11 lore raw
In reply to Ingo Oeser
Quoted 2 lines · Ingo Oeser

Stupid applications can starve other applications for a while, but not forever, because the kernel is still running and deciding.

Not so.

task 1: sched_rr, priority 1, takes mutex task 2: sched_rr, priority 2, cpu hog, infinite loop task 3: sched_rr, priority 99, tries to get mutex

And now tasks 1 and 3 are starved forever. Arguably bad application design, but it demonstrates a case where applications can starve other applications.

Chris

#12 lore raw
Quoted 8 lines · Chris Friesen
Quoted 2 lines · Ingo Oeser

Stupid applications can starve other applications for a while, but not forever, because the kernel is still running and deciding.

Not so.

Quoted 7 lines

task 1: sched_rr, priority 1, takes mutex task 2: sched_rr, priority 2, cpu hog, infinite loop task 3: sched_rr, priority 99, tries to get mutex

And now tasks 1 and 3 are starved forever. Arguably bad application design, but it demonstrates a case where applications can starve other applications.

You are right.

In "If a SCHED_RR process has been running for a time period equal to or longer than the time quantum, it will be put at the end of the list for its priority" I missed the "for its priority" part.

You would need to change the priority of task 1 until it releases the mutex. Ideally the owner gets the maximum priority of
his and all the waiters on it, until it releases his mutex, where he regains its old priority after release of mutex. But this priority elevation happens only, if he is runnable. If not, he gets his old priority back, until he is runnable.

But then again you just need to grab a mutex shared with a high priority task and consume CPU.

Since this behavior is not defined in POSIX AFAIK, you just have to write your applications properly or use SCHED_OTHER for CPU hogging.

Regards

Ingo Oeser

#13 lore raw
Quoted 9 lines · Ingo Oeser
Quoted 2 lines · Chris Friesen

Ingo Oeser wrote:
[...]

You would need to change the priority of task 1 until it releases the mutex. Ideally the owner gets the maximum priority of
his and all the waiters on it, until it releases his mutex, where he regains its old priority after release of mutex. But this priority elevation happens only, if he is runnable. If not, he gets his old priority back, until he is runnable.

This is called a "priority inversion" problem, and there was some work done by Ingo Molnar to make the scheduler aware of such cases and handle them appropriatelly.

You can follow this thread for more info:

http://marc.theaimsgroup.com/?l=linux-kernel&m=110106915415886&w=2

I really don't know what's the current state, but this is nothing new...

Signature · 4 lines
Paulo Marques - www.grupopie.com

All that is necessary for the triumph of evil is that good men do nothing.
Edmund Burke (1729 - 1797)
#14 lore raw
Quoted 12 lines · Paulo Marques
Quoted 10 lines · Ingo Oeser
Quoted 2 lines · Chris Friesen

Ingo Oeser wrote:
[...]

You would need to change the priority of task 1 until it releases the mutex. Ideally the owner gets the maximum priority of
his and all the waiters on it, until it releases his mutex, where he regains its old priority after release of mutex. But this priority elevation happens only, if he is runnable. If not, he gets his old priority back, until he is runnable.

Quoted 8 lines

This is called a "priority inversion" problem, and there was some work done by Ingo Molnar to make the scheduler aware of such cases and handle them appropriatelly.

You can follow this thread for more info:

http://marc.theaimsgroup.com/?l=linux-kernel&m=110106915415886&w=2

The solution to your problem (which is as old as the hills) involves priority inheriting mutexes which are available in the RT preempt patch (if you build with CONFIG_PREEMPT_RT). This should be usable for hard realtime applications.

http://people.redhat.com/mingo/realtime-preempt

If you just need very good soft realtime performance I recommend PREEMPT_DESKTOP.

Lee

#15 lore raw
Quoted 4 lines · Lee Revell

The solution to your problem (which is as old as the hills) involves priority inheriting mutexes which are available in the RT preempt patch (if you build with CONFIG_PREEMPT_RT). This should be usable for hard realtime applications.

Yup. I was just pointing out that userspace apps *can* block other userspace apps.

Quoted 4 lines

http://people.redhat.com/mingo/realtime-preempt

If you just need very good soft realtime performance I recommend PREEMPT_DESKTOP.

How does this compare with Inaky's "robust mutexes" patch?

Chris

#17 lore raw
In reply to Chad N. Tindel
Quoted 8 lines · Chad N. Tindel
Quoted 4 lines

If you keep a learning attitude, there is a chance for this discussion to go on. However, if you keep the "Come now, don't bullshit me, this is a broken architecture and you're just trying to cover up" attitude, you're just going to get discarded as a troll.

I'm not trying to troll here; I suppose I'm just coming from a different background. I'll try to adjust my tone.

Quoted 7 lines
Quoted 3 lines

I personally like the linux way: "root has the ability to shoot himself in the foot if he wants to". This is my computer, damn it, I am the one who tells it what to do.

I'm all for allowing people to shoot themselves in the foot. That doesn't mean that it is OK for a single userspace thread to mess up a 64-way box.

What's so special about a 64-way box?

Note that the box wasn't messed up - the thread merely used too much cpu. It is perfectly ok - even on a 64-way box - to have a thread that runs with higher priority than all the kernel threads - *�if* it occationally sleeps. That means the thread can get very low latency work done, and the kernel threads will simply wait a little. Then the thread sleeps, and those cruical kernel threads move on. A high-priority thread that doesn't run all the time is no problem. and it may need the ability to preempt kernel threads occationally due to timing constraints.

In the case mentioned, the high-priority thread ran all the time. That's bad, but there is no way the kernel can guess that is was a bad idea in that case. The kernel does what it is told. An ordinary user can�'t use such priorities, so there is no security problem here. Only root can, and root has the power to disrupt service anyway (shutdown, kill any process, delete any file.)

Someone who runs as root is _trusted_ to do the right thing, this trust might be outside the scope of the os. In other words, some people are allowed to run special processes, by the machine owner. Some gets the root password - and they are supposed to be above the "crowds" and not crash the machine just because they can.

Helge Hafting

#18 lore raw
Quoted 1 line

What's so special about a 64-way box?

They're expensive and customers don't expect a single userspace thread to tie up the other 63 CPUs no matter how buggy it is. It is intuitively obvious that a buggy kernel can bring a system to its knees, but it is not intuitively obvious that a buggy userspace app can do the same thing. It is more of a supportability issue than anything, because you expect the other processors to function properly so you can get in and live-debug the application when it hits a bug that makes it CPU-bound. This is especially important if the box is, say, in a remote jungle of China or something where you don't have access to the console.

The horse is dead, so lets not beat it anymore for the time being. It is quite clear that people don't want Linux to (by default) not have the gun cocked and pointed at the application developer's feet. People who want a kernel that doesn't hang in the face of bad-acting userspace apps can change the priority of important kernel threads, which seems like a reasonable workaround for now.

Regards,

Chad

#19 lore raw
Quoted 9 lines · Chad N. Tindel

They're expensive and customers don't expect a single userspace thread to tie up the other 63 CPUs no matter how buggy it is. It is intuitively obvious that a buggy kernel can bring a system to its knees, but it is not intuitively obvious that a buggy userspace app can do the same thing. It is more of a supportability issue than anything, because you expect the other processors to function properly so you can get in and live-debug the application when it hits a bug that makes it CPU-bound. This is especially important if the box is, say, in a remote jungle of China or something where you don't have access to the console.

"Unix policy is to not stop root from doing stupid things because that would also stop him from doing clever things." - Andi Kleen

"It's such a fine line between stupid and clever" - Derek Smalls

Lee

#20 lore raw
In reply to Chad N. Tindel
Quoted 12 lines · Chad N. Tindel
Quoted 1 line

What's so special about a 64-way box?

They're expensive and customers don't expect a single userspace thread to tie up the other 63 CPUs no matter how buggy it is. It is intuitively obvious that a buggy kernel can bring a system to its knees, but it is not intuitively obvious that a buggy userspace app can do the same thing. It is more of a supportability issue than anything, because you expect the other processors to function properly so you can get in and live-debug the application when it hits a bug that makes it CPU-bound. This is especially important if the box is, say, in a remote jungle of China or something where you don't have access to the console.

These are very good points. And the solution exists - if you want these options then simply run the program at a lower priority than the kernel threads. Doing this is not a problem.

You _can_ run a process at highest priority, but you don't have to!

Quoted 3 lines

The horse is dead, so lets not beat it anymore for the time being. It is quite clear that people don't want Linux to (by default) not have the gun cocked and pointed at the application developer's feet.

Linux is safe, and you bring up a non-issue. So what if the app couldn't get higher priority than kernel threads? You could then implement it as a kernel thread and get the same problem anyway. No difference.

Quoted 5 lines

People who want a
kernel that doesn't hang in the face of bad-acting userspace apps can change the priority of important kernel threads, which seems like a reasonable workaround for now.

Yes, or they can simply run the app at a slightly lower priority until it is fully tested so they know it can be trusted.

People sometimes need to not be delayed by kernel threads, and that is not a problem as long as the application gives up the cpu after it finishes doing the time-critical work. We want linux to be able to do these kinds of work too.

saying that the os doesn't have control does not make sense. The os will give away a cpu - but only if _you_ let it.

Helge Hafting