Linux scheduler capabilities for batch jobs.
Hello All,
I have been writing user land server code and increasingly I find myself writing resource management code which I think would be better handled by the scheduler. The problem can be described as "don't run this process if the machine is swapping." I would think that this would be a common enough need that it was already in the kernel. I'm hoping it is and I have simply overlooked it (part of the reason for this post,) but I've looked around a good bit and most of the scheduler enhancements have to do with real time and latency, not batch jobs.
My problem is analogous to a parallel make. Say I have an 8 CPU machine, and I run "make -j8". If the total memory of the 8 jobs throws the machine into swap, it begins to thrash and runtime is awful. I believe this is aggravated by the scheduler trying to be fair, and keeping all 8 processes running. If it was possible to tell the scheduler that it was OK not to be fair when scheduling these processes, I think the total runtime could be reduced if it put some of the processes to sleep while others completed. Is there a way to tell the scheduler it is allowed to do this? Should there be?
Thank You,
Hands
Quoted 5 lines · J Louis
If it was possible to tell
the scheduler that it was OK not to be fair when scheduling these processes, I think the total runtime could be reduced if it put some of the processes to sleep while others completed. Is there a way to tell the scheduler it is allowed to do this? Should there be?
There is no way to do this currently, but I suspect that it would not be too difficult to add.
Of course, if you have two tasks that are each a little larger than memory, your idea could lead to one of the processes being starved forever. This is probably not acceptable :)
In fact, one single batch process that is swapping could trigger the algorithm you described, halting itself. Your idea would need very carefuly implementation to avoid these kinds of issues, but I believe it could definately be done.
Signature · 1 line
All rights reversed.
Quoted 11 lines · Rik van Riel
Quoted 5 lines · J Louis
If it was possible to tell
the scheduler that it was OK not to be fair when scheduling these processes, I think the total runtime could be reduced if it put some of the processes to sleep while others completed. Is there a way to tell the scheduler it is allowed to do this? Should there be?There is no way to do this currently, but I suspect that it would not be too difficult to add.
Quoted 8 lines
Of course, if you have two tasks that are each a little larger than memory, your idea could lead to one of the processes being starved forever. This is probably not acceptable :)
In fact, one single batch process that is swapping could trigger the algorithm you described, halting itself. Your idea would need very carefuly implementation to avoid these kinds of issues, but I believe it could definately be done.
Some king of interaction between the swap token and the scheduler, perhaps, for SCHED_BATCH processes.
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 11 lines · Rik van Riel
Quoted 5 lines · J Louis
If it was possible to tell
the scheduler that it was OK not to be fair when scheduling these processes, I think the total runtime could be reduced if it put some of the processes to sleep while others completed. Is there a way to tell the scheduler it is allowed to do this? Should there be?There is no way to do this currently, but I suspect that it would not be too difficult to add.
Quoted 9 lines
Of course, if you have two tasks that are each a little larger than memory, your idea could lead to one of the processes being starved forever. This is probably not acceptable :)
In fact, one single batch process that is swapping could trigger the algorithm you described, halting itself. Your idea would need very carefuly implementation to avoid these kinds of issues, but I believe it could definately be done.
I think it gets doubly hard because the processes may have parts swapped even if there isn't memory pressure, so you can't just use percentage in memory. If you were going to do this in an effective way you would really need to monitor swap rate per process, and factor in total size, so it's not trivial. Of course in the extreme cases it would be hard to avoid improvement, so it need not be perfect to be helpful.
Signature · 7 lines
Bill Davidsen <[email protected]> Even purely technical things can appear to be magic, if the documentation is obscure enough. For example, PulseAudio is configured by dancing naked around a fire at midnight, shaking a rattle with one hand and a LISP manual with the other, while reciting the GNU manifesto in hexadecimal. The documentation fails to note that you must circle the fire counter-clockwise in the southern hemisphere.
Quoted 4 lines · J Louis
My problem is analogous to a parallel make. Say I have an 8 CPU machine, and I run "make -j8". If the total memory of the 8 jobs throws the machine into swap, it begins to thrash and runtime is awful.
Thrashing is more of a VM/IO scheduling concern.
Quoted 2 lines
I believe this is aggravated by the scheduler trying to be fair, and keeping all 8 processes running.
Yup, fair CPU distribution is the process scheduler's mission, and that allows tasks to compete for other resources.
Quoted 4 lines
If it was possible to tell
the scheduler that it was OK not to be fair when scheduling these processes, I think the total runtime could be reduced if it put some of the processes to sleep while others completed.
The scheduler doesn't know that any given task _ever_ completes.
Quoted 2 lines
Is there a way to
tell the scheduler it is allowed to do this? Should there be?
No, and I don't think it's feasible for existing classes. You could invent a new scheduling class, but I think you'd need to invent quite a bit of infrastructure in the VM to make it work well.
OTOH, the process scheduler doesn't, and shouldn't, make IO resource decisions, we have IO schedulers to manage who gets what IO bandwidth when. The same should apply to VM resources. Seems to me what you really want is a VM scheduler.
-Mike
Quoted 12 lines · J Louis
Hello All,
I have been writing user land server code and increasingly I find myself writing resource management code which I think would be better handled by the scheduler. The problem can be described as "don't run this process if the machine is swapping." I would think that this would be a common enough need that it was already in the kernel. I'm hoping it is and I have simply overlooked it (part of the reason for this post,) but I've looked around a good bit and most of the scheduler enhancements have to do with real time and latency, not batch jobs.
Quoted 4 lines
My problem is analogous to a parallel make. Say I have an 8 CPU machine, and I run "make -j8". If the total memory of the 8 jobs throws the machine into swap, it begins to thrash and runtime is awful. I believe this is aggravated by the scheduler trying to be
This seems to be racy by design.
make -j:
launch gcc1
gcc1 does preprocessing (10mb)
ok, still not swapping
launch gcc2
gcc2 does preprocessing (10mb)
...
gcc100 does preprocessing (10mb, 1GB total)
now gcc1..gcc100 start optimizing (100mb needed) and boom.
Pavel
Signature · 2 lines
(english) http://www.livejournal.com/~pavelmachek (cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html