Gurus, a silly question for preemptive behavior
As i know, in linux, user space application is preemptive at any time. however, linux kernel is NOT preemptive, that means, even some event is finished, Linux kernel scheduler itself still can't have opportunity to interrupt the current user application and switch it out. it is called scheduler latency. normally , the latency is about 88us in mean , maximum : 200ms. Thus, the short latency shouldn't impact user applications too much and is not a problem. It is an issue in those embedded voice processing systems by introducing jitters, thus smart people came up with two kernel schedule patch: preemptive patch and low latency patch.
My application which has nice value as 10 of low priority, however, it holds the CPU and excludes other applciations that have higher priority (nice 0) to run, my application causes CPU pegging.
Thus, I am wondering: why user space application can't be effectively interrupted? why there is CPU
pegging? Could you please educate me
on this particular issue and shed me some light to address it?
my system:
[root@sa-c2-7 proc]# uname -a
Linux sa-c2-7 2.4.21-15.ELsmp #1 SMP Thu Apr 22 00:18:24 EDT 2004 i686 i686 i386 GNU/Linux
thanks!
Quoted 7 lines · jesse
As i know, in linux, user space application is preemptive at any time. however, linux kernel is NOT preemptive, that means, even some event is finished, Linux kernel scheduler itself still can't have opportunity to interrupt the current user application and switch it out. it is called scheduler latency.
The kernel is preemptible if you enable the preempt option in the configuration. There are some code paths that are not preemptible despite this, but they are gradually being improved over time.
Quoted 7 lines
normally , the latency is about 88us in mean , maximum : 200ms. Thus, the short latency shouldn't impact user applications too much and is not a problem. It is an issue in those embedded voice processing systems by introducing jitters, thus smart people came up with two kernel schedule patch: preemptive patch and low latency patch.
You're thinking about the 2.4 kernel. 2.6 is effectively both of those patches inclusive.
Quoted 2 lines
my system: [root@sa-c2-7 proc]# uname -a Linux sa-c2-7 2.4.21-15.ELsmp #1 SMP Thu Apr 22 00:18:24 EDT 2004 i686 i686 i386 GNU/Linux
If you want lower latency on a 2.4 kernel you need further patches. You are most likely to benefit from a move to a 2.6 kernel and enabling preempt.
Cheers,
Con
Con:
thank you for your prompt reply in the holiday
season.
My point is: Even kernel 2.4 is not
preemptive, the latence should be very minimal.(<300ms)
why user space application with low nice priority can't be effectively interrupted and holds the CPU resource since all user space application is preemptive?
Merry Xmas.
jesse
Quoted 52 lines · --- Con Kolivas
Quoted 3 lines · jesse
As i know, in linux, user space application is preemptive at any time. however, linux kernel is
NOT
Quoted 1 line
preemptive, that means, even some event is
finished,
Quoted 2 lines
Linux kernel scheduler itself still can't have opportunity to interrupt the current user
application
Quoted 1 line
and switch it out. it is called scheduler latency.
The kernel is preemptible if you enable the preempt option in the
configuration. There are some code paths that are not preemptible despite
this, but they are gradually being improved over time.Quoted 1 line
normally , the latency is about 88us in mean ,
maximum
Quoted 1 line
: 200ms. Thus, the short latency shouldn't impact
user
Quoted 1 line
applications too much and is not a problem. It is
an
Quoted 1 line
issue in those embedded voice processing systems
by
Quoted 1 line
introducing jitters, thus smart people came up
with
Quoted 1 line
two kernel schedule patch: preemptive patch and
low
Quoted 1 line
latency patch.
You're thinking about the 2.4 kernel. 2.6 is effectively both of those
patches inclusive.Quoted 4 lines
my system:
[root@sa-c2-7 proc]# uname -a
Linux sa-c2-7 2.4.21-15.ELsmp #1 SMP Thu Apr 22 00:18:24 EDT 2004 i686 i686 i386 GNU/LinuxIf you want lower latency on a 2.4 kernel you need further patches. You are
most likely to benefit from a move to a 2.6 kernel and enabling preempt.Cheers,
Con
Quoted 5 lines · jesse
Con:
thank you for your prompt reply in the holiday
season.
Quoted 7 lines
My point is: Even kernel 2.4 is not
preemptive, the latence should be very minimal.(<300ms)
why user space application with low nice priority can't be effectively interrupted and holds the CPU resource since all user space application is preemptive?
If your process has got work to do and has a higher priority than other processes, it gets to run. If you don't want this behavior, don't give it such a high priority.
If you want low latency to do some quick high priority task, just do it quickly and relinquish the processor, instead of hogging it.
What are you trying to accomplish, anyway?
Signature · 4 lines
Paulo Marques - www.grupopie.com "A journey of a thousand miles begins with a single step." Lao-tzu, The Way of Lao-tzu
Paulo:
I already said in the messsage that my user space
application has a low nice priority, i set it to 10. since my application has low priority compared to other user space applications, it is supposed to be interrupted. but it is not.
regards!
jesse
Quoted 34 lines · --- Paulo Marques
Quoted 5 lines · jesse
Con:
thank you for your prompt reply in the holiday
season.Quoted 7 lines
My point is: Even kernel 2.4 is not
preemptive, the latence should be very minimal.(<300ms)
why user space application with low nice priority can't be effectively interrupted and holds the CPU resource since all user space application is preemptive?If your process has got work to do and has a higher priority than other
processes, it gets to run. If you don't want this behavior, don't give
it such a high priority.If you want low latency to do some quick high
priority task, just do it
quickly and relinquish the processor, instead of
hogging it.What are you trying to accomplish, anyway?
Signature · 5 lines
Paulo Marques - www.grupopie.com "A journey of a thousand miles begins with a single step." Lao-tzu, The Way of Lao-tzu
Quoted 7 lines · jesse
Paulo:
I already said in the messsage that my user space
application has a low nice priority, i set it to 10. since my application has low priority compared to other user space applications, it is supposed to be interrupted. but it is not.
The confusion comes from "low nice priority". The lower the nice value the higher the priority.
Anyway, you still haven't give enough data to analyze. What does your application do? Is it I/O intensive? If it is, it could be that the kernel itself is hogging the CPU doing I/O on behalf of a low prio process (priority, specially in 2.4, only affects CPU distribution and not I/O). How do you know it's not being preempted? What is your .config? What patches do you have applied? And finally, why don't you upgrade to a 2.6 kernel :) ?
Signature · 4 lines
Paulo Marques - www.grupopie.com "A journey of a thousand miles begins with a single step." Lao-tzu, The Way of Lao-tzu
Quoted 7 lines · jesse
Paulo:
I already said in the messsage that my user space
application has a low nice priority, i set it to 10. since my application has low priority compared to other user space applications, it is supposed to be interrupted. but it is not.
If your task is better priority the scheduler will make it preempt the worse priority task. It sounds to me like you are complaining that the worse priority task is still getting cpu? If so, you misunderstand priority - it orders tasks according to priority giving lower latency and preemptive behaviour to the better task, and gives _more_ cpu but not all the cpu. The cpu must still be shared, but with more cpu distributed to the better priority task. If you want your better priority task to get _all_ the cpu you have to use real time scheduling.
Cheers,
Con
P.S. Please dont top post when replying, it's hard to follow an email thread to see what you're replying to if you do that. These threads can get very long and confusing.
Quoted 29 lines · --- Con Kolivas
Quoted 3 lines · jesse
Paulo:
I already said in the messsage that my user
space
Quoted 1 line
application has a low nice priority, i set it to
10.
Quoted 2 lines
since my application has low priority compared to
other user space applications, it is supposed tobe
Quoted 1 line
interrupted. but it is not.
If your task is better priority the scheduler will make it preempt the
worse priority task. It sounds to me like you are complaining that the
worse priority task is still getting cpu? If so, you misunderstand
priority - it orders tasks according to priority giving lower latency
and preemptive behaviour to the better task, and gives _more_ cpu but
not all the cpu. The cpu must still be shared, but with more cpu
distributed to the better priority task. If you want your better
priority task to get _all_ the cpu you have to use real time scheduling.
Quoted 3 lines
Cheers,
Con
ok, Con, your explaining makes some sense to me , but still not very well.
suppose I have five high process: A1, A2, A3, A4,
A5 all have nice = 0. and I also have a low priority process B with nice = 10.
a) when process B is scheduled to run, it is given
a short time slot based on its priority, for example 5 secs. because at that point, A1/2/3/4/5 are not started yet. B will get CPU and run at full speed.
b) at the end of time slot(5 secs), scheduler
finds higher priority A1/A2/A3/A4/A5 are ready, scheduler will interrupt process B and starts to pick a process from group A, even though B still needs CPU cycle.
c)unfortunately, process A1/2/3/4/5 are so active,
thus process B should never get opportunity to run again, in consequence, CPU Usage% of Process B should be very Low.
However, The above theretic assumption is in
contrary to what i observed. in my LAB, the low priority process B seems to hold the CPU forever and Top command always shows Process B with a 90% CPU usage.
If _more_ cpu but not _all_ the cpu are given to
Process A1/2/3/4/5, Process B shouldn't have a 90% CPU usage.
Thus, i can't help asking why low priority process B
gets most CPU cycle.
thanks in advance.
jesse
Quoted 55 lines · jesse
Quoted 22 lines · --- Con Kolivas
Quoted 3 lines · jesse
Paulo:
I already said in the messsage that my user
space
Quoted 1 line
application has a low nice priority, i set it to
10.
Quoted 2 lines
since my application has low priority compared to
other user space applications, it is supposed tobe
Quoted 1 line
interrupted. but it is not.
If your task is better priority the scheduler will
make it preempt the worse priority task. It sounds to me like you are complaining that the worse priority task is still getting cpu? If so, you misunderstand priority - it orders tasks according to priority giving lower latency and preemptive behaviour to the better task, and gives _more_ cpu but not all the cpu. The cpu must still be shared, but with more cpu distributed to the better priority task. If you want your better priority task to get _all_ the cpu you have to use real time scheduling.Quoted 3 lines
Cheers,
Conok, Con, your explaining makes some sense to me , but still not very well.
suppose I have five high process: A1, A2, A3, A4,
A5 all have nice = 0. and I also have a low priority process B with nice = 10.a) when process B is scheduled to run, it is given
a short time slot based on its priority, for example 5
secs. because at that point, A1/2/3/4/5 are not
started yet. B will get CPU and run at full speed.
b) at the end of time slot(5 secs), scheduler
finds higher priority A1/A2/A3/A4/A5 are ready,
scheduler will interrupt process B and starts to pick a process from group A, even though B still needs CPU cycle.
c)unfortunately, process A1/2/3/4/5 are so active,
thus process B should never get opportunity to run
again, in consequence, CPU Usage% of Process B should
be very Low.However, The above theretic assumption is in
contrary to what i observed. in my LAB, the low priority process B seems to hold the CPU forever and Top command always shows Process B with a 90% CPU usage.
Quoted 6 lines
If _more_ cpu but not _all_ the cpu are given to
Process A1/2/3/4/5, Process B shouldn't have a 90% CPU usage.Thus, i can't help asking why low priority process B
gets most CPU cycle.
What you are describing is completely wrong behaviour. Please post output of top running during this workload to demonstrate/prove this is happening. Easiest thing to do is get your workload running and do 'top -b -n 1 > top.log' and post top.log please.
Cheers,
Con
Quoted 63 lines · jesse
Quoted 29 lines · --- Con Kolivas
Quoted 3 lines · jesse
Paulo:
I already said in the messsage that my user
space
Quoted 1 line
application has a low nice priority, i set it to
10.
Quoted 2 lines
since my application has low priority compared to
other user space applications, it is supposed tobe
Quoted 1 line
interrupted. but it is not.
If your task is better priority the scheduler will
make it preempt the worse priority task. It sounds to me like you are complaining that the worse priority task is still getting cpu? If so, you misunderstand priority - it orders tasks according to priority giving lower latency and preemptive behaviour to the better task, and gives _more_ cpu but not all the cpu. The cpu must still be shared, but with more cpu distributed to the better priority task. If you want your better priority task to get _all_ the cpu you have to use real time scheduling.Quoted 3 lines
Cheers,
Conok, Con, your explaining makes some sense to me , but still not very well.
suppose I have five high process: A1, A2, A3, A4,
A5 all have nice = 0. and I also have a low priority process B with nice = 10.a) when process B is scheduled to run, it is given
a short time slot based on its priority, for example 5
secs. because at that point, A1/2/3/4/5 are not
started yet. B will get CPU and run at full speed.
b) at the end of time slot(5 secs), scheduler
finds higher priority A1/A2/A3/A4/A5 are ready,
scheduler will interrupt process B and starts to pick a process from group A, even though B still needs CPU cycle.
c)unfortunately, process A1/2/3/4/5 are so active,
thus process B should never get opportunity to run
again, in consequence, CPU Usage% of Process B should
be very Low.However, The above theretic assumption is in
contrary to what i observed. in my LAB, the low priority process B seems to hold the CPU forever and Top command always shows Process B with a 90% CPU usage.
Quoted 3 lines
If _more_ cpu but not _all_ the cpu are given to
Process A1/2/3/4/5, Process B shouldn't have a 90% CPU usage.
This statement is not exactly true. If process B is always ready to run (CPU bound, not I/O bound) it could easily consume more CPU than all the other processes combined. It all depends on what A1/2/3/4/5 are doing. Just because A1 has a higher priority doesn't mean it is ready to run. If processes A1/2/3/4/5 spend most of their time waiting for I/O or sleeping (not ready to run) and B does a lot of computation or just busy spins, B could easily consume more CPU than the others.
Quoted 16 lines
Thus, i can't help asking why low priority process B
gets most CPU cycle.thanks in advance.
jesse
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Quoted 20 lines · jesse
As i know, in linux, user space application is preemptive at any time. however, linux kernel is NOT preemptive, that means, even some event is finished, Linux kernel scheduler itself still can't have opportunity to interrupt the current user application and switch it out. it is called scheduler latency. normally , the latency is about 88us in mean , maximum : 200ms. Thus, the short latency shouldn't impact user applications too much and is not a problem. It is an issue in those embedded voice processing systems by introducing jitters, thus smart people came up with two kernel schedule patch: preemptive patch and low latency patch.
My application which has nice value as 10 of low priority, however, it holds the CPU and excludes other applciations that have higher priority (nice 0) to run, my application causes CPU pegging.
Quoted 7 lines
Thus, I am wondering: why user space application can't be effectively interrupted? why there is CPU
pegging? Could you please educate me on this particular issue and shed me some light to
address it?my system: [root@sa-c2-7 proc]# uname -a Linux sa-c2-7 2.4.21-15.ELsmp #1 SMP Thu Apr 22 00:18:24 EDT 2004 i686 i686 i386 GNU/Linux
First, a pre-emption happens when an external event occurs that makes a higher priority task change state from blocked to ready-to-run, and the scheduling algorithm says that the new task should be run in place of the old task -- and the complexity of the time-slice schedule is such that your assumption doesn't always hold true.
If you have a low-priority task that needs to make sure that higher-priority functions *always* run, and you can't move up to the 2.6 kernel (or even after moving up you find yourself with problems) the userland solution is a simple one:
if (1) {
struct timeval delay = {0,20000};
select(0, NULL, NULL, NULL, &delay);
}(DON'T declare the variable delay as constant, because select modifies the contents of this variable.)
A number of real-time operating systems provide for a give-up-control system call, which says "if you have something better to do than to run me, then do it" for just this sort of thing. The legacy of the Linux schedules being from a classic time-sharing system, which trys to implement "fairness" in allocating CPU resources, there is no equivalent call that I'm aware of.
So you have to make your program block in order to guarantee that you give up the CPU. The delay of 20 ms in my example call assumes your HZ value is 100, and that you really don't mind having a few idle CPU cycles if your higher-priority program isn't ready to run when you make the select call. In my Web servers, I use 100 ms delays to ensure that I/O for other processes is allowed to take place; Webalizer stats, for example, have hogged disk access to the point that it impacts the entire system, so I've inserted the Perl equivalent of the above code in key places so that background tasks really do run in the background.
A delay value of zero to select is, I believe, an effective no-operation which may or may not trigger a scheduling event. (I looked a long time ago at this -- many, MANY kernels ago -- and it's as though you didn't make the call at all, absent the CPU cycles expended to make the call and for the kernel to get back to the userland task.) A value of 10 ms (on a system with HZ set to 100) can also be taken as though you coded zero in some cases -- at least some code I wrote a couple of years ago suggests this.
Why select? I write portable code, and this is in my cookbook. There is a nanosleep system call that is part of POSIX, and that looks to be a preferable way to deal with things. You would need to look at the Linux source code to see what a nanosleep with a value of one nanosecond does, and whether that's guaranteed to block your code for one scheduling cycle.
How frequently you put scheduling breaks into your code is up to you. I typicall profile code, and try to put in scheduling breaks at 1/4 of the desired system response latency. For a 100 ms latency, that means that I do scheduling breaks no less frequently than 25 ms., and in many cases more often than that. Don't forget that I/O provides breaks, too, so the number of places where you need to put in explicit scheduling breaks may be small, indeed.
For Webalizer, I run a scheduling break every 100 lines read from the log files. That count, by experimentation, has shown to provide a good balance between throughout of the background process and system reaction latency. That's stuff running on a 2.4 kernel, by the way, because I'm slow to adopt new, bleeding-edge stuff in my production environments.
A couple of thoughts from a different place in the trenches.
Stephen Satchell