2.6.0 batch scheduling, HT aware
PATCH 1/1 2.6.0 batch scheduling, HT aware
I've done a resync and update of my batch scheduling that is also hyper-thread aware.
What is batch scheduling? Specifying a task as batch allows it to only use cpu time if there is idle time available, rather than having a proportion of the cpu time based on niceness.
Why do I need hyper-thread aware batch scheduling?
If you have a hyperthread (P4HT) processor and run it as two logical cpus you can have a very low priority task running that can consume 50% of your physical cpu's capacity no matter how high priority tasks you are running. For example if you use the distributed computing client setiathome you will be effectively be running at half your cpu's speed even if you run setiathome at nice 20. Batch scheduling for normal cpus allows only idle time to be used for batch tasks, and for HT cpus only allows idle time when both logical cpus are idle.
This is not being pushed for mainline kernel inclusion, but the issue of how to prevent low priority tasks slowing down HT cpus needs to be considered for the mainline HT scheduler if it ever gets included. This patch provides a temporising measure for those with HT processors, and a demonstrative way to handle them in mainline.
Patch available here:
http://ck.kolivas.org/patches/2.6/2.6.0/
Con
Quoted 10 lines · Con Kolivas
I've done a resync and update of my batch scheduling that is also hyper-thread aware.
What is batch scheduling? Specifying a task as batch allows it to only use cpu time if there is idle time available, rather than having a proportion of the cpu time based on niceness.
Why do I need hyper-thread aware batch scheduling?
If you have a hyperthread (P4HT) processor and run it as two logical cpus you can have a very low priority task running that can consume 50% of your physical cpu's capacity no matter how high priority tasks you are running. For example if you use the distributed computing client setiathome you will be effectively be running at half your cpu's speed even if you run setiathome at nice 20. Batch scheduling for normal cpus allows only idle time to be used for batch tasks, and for HT cpus only allows idle time when both logical cpus are idle.
This is not being pushed for mainline kernel inclusion, but the issue of how to prevent low priority tasks slowing down HT cpus needs to be considered for the mainline HT scheduler if it ever gets included. This patch provides a temporising measure for those with HT processors, and a demonstrative way to handle them in mainline.
I wonder how does Intel suggest we handle this problem? Batch scheduling aside, I wonder how to do any sort of priorities at all? I think POWER5 can do priorities in hardware, that is the only sane way I can think of doing it.
I think this patch is much too ugly to get into such an elegant scheduler. No fault to you Con because its an ugly problem.
How about this: if a task is "delta" priority points below a task running on another sibling, move it to that sibling (so priorities via timeslice start working). I call it active unbalancing! I might be able to make it fit if there is interest. Other suggestions?
Quoted 2 lines · Nick Piggin
I think this patch is much too ugly to get into such an elegant scheduler. No fault to you Con because its an ugly problem.
You're too kind. No it's ugly because of my code but it works for now.
Quoted 4 lines
How about this: if a task is "delta" priority points below a task running on another sibling, move it to that sibling (so priorities via timeslice start working). I call it active unbalancing! I might be able to make it fit if there is interest. Other suggestions?
I discussed this with Ingo and that's the sort of thing we thought of. Perhaps a relative crossover of 10 dynamic priorities and an absolute crossover of 5 static priorities before things got queued together. This is really only required for the UP HT case.
Con
Quoted 8 lines · Con Kolivas
Quoted 3 lines · Nick Piggin
I think this patch is much too ugly to get into such an elegant scheduler. No fault to you Con because its an ugly problem.
You're too kind. No it's ugly because of my code but it works for now.
Well its all the special cases for batch scheduling that I don't like, the idea to not run batch tasks on a package running non batch processes is sound. I thought the batch scheduling code is Ingo's, but I could be mistaken. Anyway...
Quoted 9 lines
Quoted 6 lines
How about this: if a task is "delta" priority points below a task running on another sibling, move it to that sibling (so priorities via timeslice start working). I call it active unbalancing! I might be able to make it fit if there is interest. Other suggestions?
I discussed this with Ingo and that's the sort of thing we thought of. Perhaps a relative crossover of 10 dynamic priorities and an absolute crossover of 5 static priorities before things got queued together. This is really only required for the UP HT case.
Well I guess it would still be nice for "SMP HT" as well. Hopefully the code can be generic enough that it would just carry over nicely. It does have complications though because the load balancer would have to be taught about it, and those architectures that do hardware priorities probably don't even want it.
Quoted 11 lines · Nick Piggin
Quoted 5 lines · Con Kolivas
Quoted 2 lines · Nick Piggin
I think this patch is much too ugly to get into such an elegant scheduler. No fault to you Con because its an ugly problem.
You're too kind. No it's ugly because of my code but it works for now.
Well its all the special cases for batch scheduling that I don't like, the idea to not run batch tasks on a package running non batch processes is sound. I thought the batch scheduling code is Ingo's, but I could be mistaken. Anyway...
I realise the special cases suck. Code for one setting in a spot where it affects everyone is bad. Regarding the batch scheduling; no that's my special flavour coded ugly from the ground up. Ingo's is much smarter than this but once again I needed something that works now without too much effort.
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines
How about this: if a task is "delta" priority points below a task running on another sibling, move it to that sibling (so priorities via timeslice start working). I call it active unbalancing! I might be able to make it fit if there is interest. Other suggestions?
Quoted 7 lines
Quoted 4 lines
I discussed this with Ingo and that's the sort of thing we thought of. Perhaps a relative crossover of 10 dynamic priorities and an absolute crossover of 5 static priorities before things got queued together. This is really only required for the UP HT case.
Well I guess it would still be nice for "SMP HT" as well. Hopefully the code can be generic enough that it would just carry over nicely.
I disagree. I can't think of a real world scenario where 2+ physical cpus would benefit from this.
Quoted 4 lines
It does
have complications though because the load balancer would have to be taught about it, and those architectures that do hardware priorities probably don't even want it.
Probably the simple relative/absolute will have to suffice. However it still doesn't help the fact that running something cpu bound concurrently at nice 0 with something interactive nice 0 is actually slower if you use a UP HT processor in SMP mode instead of UP.
Con
Quoted 9 lines · Con Kolivas
Quoted 7 lines · Nick Piggin
Quoted 5 lines · Con Kolivas
I discussed this with Ingo and that's the sort of thing we thought of. Perhaps a relative crossover of 10 dynamic priorities and an absolute crossover of 5 static priorities before things got queued together. This is really only required for the UP HT case.
Quoted 6 lines
Quoted 3 lines
Well I guess it would still be nice for "SMP HT" as well. Hopefully the code can be generic enough that it would just carry over nicely.
I disagree. I can't think of a real world scenario where 2+ physical cpus would benefit from this.
Well its the same problem. A nice -20 process can still lose 40-55% of its performance to a nice 19 process, a figure of 10% is probably too high and we'd really want it <= 5% like what happens with a single logical processor.
Quoted 8 lines
Quoted 5 lines
It does have complications though because the load balancer would have to be taught about it, and those architectures that do hardware priorities probably don't even want it.
Probably the simple relative/absolute will have to suffice. However it still doesn't help the fact that running something cpu bound concurrently at nice 0 with something interactive nice 0 is actually slower if you use a UP HT processor in SMP mode instead of UP.
It will be based on dynamic priorities, possibly with some feedback from nice as well, but it probably still won't be perfect and it will probably be very complex *cough* hardware priorities *cough* ;)
I might try to fit it into a more general priority balancing system because we currently have similar sorts of failings on regular SMP as well.
Quoted 11 lines · Nick Piggin
Quoted 10 lines · Con Kolivas
Quoted 8 lines · Nick Piggin
Quoted 4 lines · Con Kolivas
I discussed this with Ingo and that's the sort of thing we thought of. Perhaps a relative crossover of 10 dynamic priorities and an absolute crossover of 5 static priorities before things got queued together. This is really only required for the UP HT case.
Well I guess it would still be nice for "SMP HT" as well. Hopefully the code can be generic enough that it would just carry over nicely.
Quoted 7 lines
Quoted 2 lines
I disagree. I can't think of a real world scenario where 2+ physical cpus would benefit from this.
Well its the same problem. A nice -20 process can still lose 40-55% of its performance to a nice 19 process, a figure of 10% is probably too high and we'd really want it <= 5% like what happens with a single logical processor.
I changed my mind just after I sent that mail. 4 physical cores running three nice 20 and one nice -20 task gives the nice -20 task only 25% of the total cpu and 25% to each of the nice 20 tasks.
Quoted 10 lines
Quoted 9 lines
Quoted 4 lines
It does
have complications though because the load balancer would have to be taught about it, and those architectures that do hardware priorities probably don't even want it.Probably the simple relative/absolute will have to suffice. However it still doesn't help the fact that running something cpu bound concurrently at nice 0 with something interactive nice 0 is actually slower if you use a UP HT processor in SMP mode instead of UP.
Quoted 6 lines
It will be based on dynamic priorities, possibly with some feedback from nice as well, but it probably still won't be perfect and it will probably be very complex *cough* hardware priorities *cough* ;)
I might try to fit it into a more general priority balancing system because we currently have similar sorts of failings on regular SMP as well.
I'll keep my eyes peeled. Meanwhile I'll use my ugly patch ;-)
Con
Quoted 15 lines · Con Kolivas
Quoted 14 lines · Nick Piggin
Quoted 12 lines · Con Kolivas
Quoted 8 lines · Nick Piggin
Quoted 4 lines · Con Kolivas
I discussed this with Ingo and that's the sort of thing we thought of. Perhaps a relative crossover of 10 dynamic priorities and an absolute crossover of 5 static priorities before things got queued together. This is really only required for the UP HT case.
Well I guess it would still be nice for "SMP HT" as well. Hopefully the code can be generic enough that it would just carry over nicely.
I disagree. I can't think of a real world scenario where 2+ physical cpus would benefit from this.
Quoted 8 lines
Quoted 4 lines
Well its the same problem. A nice -20 process can still lose 40-55% of its performance to a nice 19 process, a figure of 10% is probably too high and we'd really want it <= 5% like what happens with a single logical processor.
I changed my mind just after I sent that mail. 4 physical cores running three nice 20 and one nice -20 task gives the nice -20 task only 25% of the total cpu and 25% to each of the nice 20 tasks.
Err that should read 4 logical cores.
Con
In article <[email protected]>,
Nick Piggin <[email protected]> wrote:
| | | Con Kolivas wrote: | | >On Tue, 23 Dec 2003 12:36, Nick Piggin wrote: | > | >>Con Kolivas wrote: | >> | >>>I discussed this with Ingo and that's the sort of thing we thought of. | >>>Perhaps a relative crossover of 10 dynamic priorities and an absolute | >>>crossover of 5 static priorities before things got queued together. This | >>>is really only required for the UP HT case.
There are two goals here. Not having a batch process on one siling makes sense, and I'm going to try Con's patch after I try Nick's latest. Actually, if they play nicely I would use both, batch would be very useful for nightly report generation on servers.
But WRT the whole HT scheduling, it would seem that ideally you want to schedule the two (or N) processes which have the lowest aggregate cache thrash, if you had a way to determine that. I suspect that a process which had a small itterative inner loop with a code+data footprint of 2-3k would coexist well with almost anything else. Minimizing the FPU contention also would improve performance, no doubt. I don't know that there are the tools at the moment to get this information, but it seems as though until it's available any scheduling will be working in the dark to some extent.
Feel free to tell me I misread this problem.
| >>> | >>Well I guess it would still be nice for "SMP HT" as well. Hopefully the | >>code can be generic enough that it would just carry over nicely. | >> | > | >I disagree. I can't think of a real world scenario where 2+ physical cpus | >would benefit from this. | > | | Well its the same problem. A nice -20 process can still lose 40-55% of its | performance to a nice 19 process, a figure of 10% is probably too high and | we'd really want it <= 5% like what happens with a single logical processor. | | > | >>It does | >>have complications though because the load balancer would have to be taught | >>about it, and those architectures that do hardware priorities probably | >>don't even want it. | >> | > | >Probably the simple relative/absolute will have to suffice. However it still | >doesn't help the fact that running something cpu bound concurrently at nice 0 | >with something interactive nice 0 is actually slower if you use a UP HT | >processor in SMP mode instead of UP. | > | | It will be based on dynamic priorities, possibly with some feedback from | nice as well, but it probably still won't be perfect and it will probably | be very complex *cough* hardware priorities *cough* ;) | | I might try to fit it into a more general priority balancing system because | we currently have similar sorts of failings on regular SMP as well.
I my experience, on servers it's more important to avoid really bad behaviour all of the time than to have perfect behaviour most of the time. All of the recent scheduler work from Nick, Con and Ingo has avoided "jackpot cases" quite well, for which I thank you and encourage you to continue. If server response goes from 20ms to 100ms Saturday night, we discuss it at a status meeting Monday morning and make suggestions to management. If response goes to 2sec we discuss it with management at 2am and they make suggestions :-(
So far 2.6.0 has been quite good at "bend but do not break" under load. Great job!
Signature · 3 lines
bill davidsen <[email protected]> CTO, TMR Associates, Inc Doing interesting things with little computers since 1979.
Quoted 4 lines · bill davidsen
There are two goals here. Not having a batch process on one siling makes sense, and I'm going to try Con's patch after I try Nick's latest. Actually, if they play nicely I would use both, batch would be very useful for nightly report generation on servers.
No hope of them playing nicely, but at some later stage I might resync on top of Nick's work if I like the direction it takes (which looks likely!)
Quoted 9 lines
But WRT the whole HT scheduling, it would seem that ideally you want to schedule the two (or N) processes which have the lowest aggregate cache thrash, if you had a way to determine that. I suspect that a process which had a small itterative inner loop with a code+data footprint of 2-3k would coexist well with almost anything else. Minimizing the FPU contention also would improve performance, no doubt. I don't know that there are the tools at the moment to get this information, but it seems as though until it's available any scheduling will be working in the dark to some extent.
Impossible with current tools. Only userspace would have a chance of predicting this and the simple rule we work off is that userspace can't be trusted so this does not appear doable in the foreseeable future.
Quoted 11 lines
Feel free to tell me I misread this problem.
I my experience, on servers it's more important to avoid really bad behaviour all of the time than to have perfect behaviour most of the time. All of the recent scheduler work from Nick, Con and Ingo has avoided "jackpot cases" quite well, for which I thank you and encourage you to continue. If server response goes from 20ms to 100ms Saturday night, we discuss it at a status meeting Monday morning and make suggestions to management. If response goes to 2sec we discuss it with management at 2am and they make suggestions :-(
Quoted 2 lines
So far 2.6.0 has been quite good at "bend but do not break" under load. Great job!
Excellent! I'm sure we'll hear from you when you turn the knob up to 11/10.
Con
Hi!
Quoted 16 lines
Quoted 16 lines
Quoted 15 lines
Quoted 10 lines
Quoted 7 lines
Quoted 4 lines
I discussed this with Ingo and that's the sort of thing we thought of. Perhaps a relative crossover of 10 dynamic priorities and an absolute crossover of 5 static priorities before things got queued together. This is really only required for the UP HT case.
Well I guess it would still be nice for "SMP HT" as well. Hopefully the code can be generic enough that it would just carry over nicely.
I disagree. I can't think of a real world scenario where 2+ physical cpus would benefit from this.
Well its the same problem. A nice -20 process can still lose 40-55% of its performance to a nice 19 process, a figure of 10% is probably too high and we'd really want it <= 5% like what happens with a single logical processor.
Quoted 5 lines
Quoted 3 lines
I changed my mind just after I sent that mail. 4 physical cores running three nice 20 and one nice -20 task gives the nice -20 task only 25% of the total cpu and 25% to each of the nice 20 tasks.
Err that should read 4 logical cores.
Actually, for 4 physical cores it is going to be true, too. And if you are memory-bound, stopping those 3 task can speed your important task, too. Its really same.
Pavel
Signature · 2 lines
When do you have a heart between your knees? [Johanka's followup: and *two* hearts?]
Hi!
Quoted 9 lines
I've done a resync and update of my batch scheduling that is also hyper-thread aware.
What is batch scheduling? Specifying a task as batch allows it to only use cpu time if there is idle time available, rather than having a proportion of the cpu time based on niceness.
Why do I need hyper-thread aware batch scheduling?
Quoted 8 lines
If you have a hyperthread (P4HT) processor and run it as two logical cpus you can have a very low priority task running that can consume 50% of your physical cpu's capacity no matter how high priority tasks you are running. For example if you use the distributed computing client setiathome you will be effectively be running at half your cpu's speed even if you run setiathome at nice 20. Batch scheduling for normal cpus allows only idle time to be used for batch tasks, and for HT cpus only allows idle time when both logical cpus are idle.
BTW this is going to be an issue even on normal (non-HT) systems. Imagine memory-bound scientific task on CPU0 and nice -20 memory-bound seti&home at CPU1. Even without hyperthreading, your scientific task is going to run at 50% of speed and seti&home is going to get second half. Oops.
Something similar can happen with disk, but we are moving out of cpu-scheduler arena with that.
[I do not have SMP nearby to demonstrate it, anybody wanting to benchmark a bit?]
Pavel
Signature · 2 lines
When do you have a heart between your knees? [Johanka's followup: and *two* hearts?]
Quoted 26 lines · Pavel Machek
Hi!
Quoted 9 lines
I've done a resync and update of my batch scheduling that is also hyper-thread aware.
What is batch scheduling? Specifying a task as batch allows it to only use cpu time if there is idle time available, rather than having a proportion of the cpu time based on niceness.
Why do I need hyper-thread aware batch scheduling?
Quoted 8 lines
If you have a hyperthread (P4HT) processor and run it as two logical cpus you can have a very low priority task running that can consume 50% of your physical cpu's capacity no matter how high priority tasks you are running. For example if you use the distributed computing client setiathome you will be effectively be running at half your cpu's speed even if you run setiathome at nice 20. Batch scheduling for normal cpus allows only idle time to be used for batch tasks, and for HT cpus only allows idle time when both logical cpus are idle.
BTW this is going to be an issue even on normal (non-HT) systems. Imagine memory-bound scientific task on CPU0 and nice -20 memory-bound seti&home at CPU1. Even without hyperthreading, your scientific task is going to run at 50% of speed and seti&home is going to get second half. Oops.
Quoted 5 lines
Something similar can happen with disk, but we are moving out of cpu-scheduler arena with that.
[I do not have SMP nearby to demonstrate it, anybody wanting to benchmark a bit?]
This is definitely the case but there is one huge difference. If you have 2x1Ghz non HT processors then the fastest a single threaded task can run is at 1Ghz. If you have 1x2Ghz HT processor the fastest a single threaded task can run is 2Ghz.
Con
Quoted 30 lines · Con Kolivas
Quoted 29 lines · Pavel Machek
Hi!
Quoted 9 lines
I've done a resync and update of my batch scheduling that is also hyper-thread aware.
What is batch scheduling? Specifying a task as batch allows it to only use cpu time if there is idle time available, rather than having a proportion of the cpu time based on niceness.
Why do I need hyper-thread aware batch scheduling?
Quoted 8 lines
If you have a hyperthread (P4HT) processor and run it as two logical cpus you can have a very low priority task running that can consume 50% of your physical cpu's capacity no matter how high priority tasks you are running. For example if you use the distributed computing client setiathome you will be effectively be running at half your cpu's speed even if you run setiathome at nice 20. Batch scheduling for normal cpus allows only idle time to be used for batch tasks, and for HT cpus only allows idle time when both logical cpus are idle.
BTW this is going to be an issue even on normal (non-HT) systems. Imagine memory-bound scientific task on CPU0 and nice -20 memory-bound seti&home at CPU1. Even without hyperthreading, your scientific task is going to run at 50% of speed and seti&home is going to get second half. Oops.
Something similar can happen with disk, but we are moving out of cpu-scheduler arena with that.
Quoted 7 lines
Quoted 2 lines
[I do not have SMP nearby to demonstrate it, anybody wanting to benchmark a bit?]
This is definitely the case but there is one huge difference. If you have 2x1Ghz non HT processors then the fastest a single threaded task can run is at 1Ghz. If you have 1x2Ghz HT processor the fastest a single threaded task can run is 2Ghz.
Or even if you have 1024x1Ghz cpus. One thread can still only run at 1Ghz. Sure there are other things that can run on other cpus which will allow the single thread to run unabated at 1Ghz, but no faster.
Con
Quoted 20 lines · Pavel Machek
Hi!
Quoted 9 lines
I've done a resync and update of my batch scheduling that is also hyper-thread aware.
What is batch scheduling? Specifying a task as batch allows it to only use cpu time if there is idle time available, rather than having a proportion of the cpu time based on niceness.
Why do I need hyper-thread aware batch scheduling?
If you have a hyperthread (P4HT) processor and run it as two logical cpus you can have a very low priority task running that can consume 50% of your physical cpu's capacity no matter how high priority tasks you are running. For example if you use the distributed computing client setiathome you will be effectively be running at half your cpu's speed even if you run setiathome at nice 20. Batch scheduling for normal cpus allows only idle time to be used for batch tasks, and for HT cpus only allows idle time when both logical cpus are idle.
BTW this is going to be an issue even on normal (non-HT) systems. Imagine memory-bound scientific task on CPU0 and nice -20 memory-bound seti&home at CPU1. Even without hyperthreading, your scientific task is going to run at 50% of speed and seti&home is going to get second half. Oops.
Quoted 7 lines
Something similar can happen with disk, but we are moving out of cpu-scheduler arena with that.
[I do not have SMP nearby to demonstrate it, anybody wanting to benchmark a bit?]
Pavel
heh...and the situation gets even worse when you add cpus, with 16way you get only 1/16 of the speed ;)
--Mika
Hi!
Quoted 27 lines
Quoted 27 lines
Quoted 9 lines
I've done a resync and update of my batch scheduling that is also hyper-thread aware.
What is batch scheduling? Specifying a task as batch allows it to only use cpu time if there is idle time available, rather than having a proportion of the cpu time based on niceness.
Why do I need hyper-thread aware batch scheduling?
Quoted 8 lines
If you have a hyperthread (P4HT) processor and run it as two logical cpus you can have a very low priority task running that can consume 50% of your physical cpu's capacity no matter how high priority tasks you are running. For example if you use the distributed computing client setiathome you will be effectively be running at half your cpu's speed even if you run setiathome at nice 20. Batch scheduling for normal cpus allows only idle time to be used for batch tasks, and for HT cpus only allows idle time when both logical cpus are idle.
BTW this is going to be an issue even on normal (non-HT) systems. Imagine memory-bound scientific task on CPU0 and nice -20 memory-bound seti&home at CPU1. Even without hyperthreading, your scientific task is going to run at 50% of speed and seti&home is going to get second half. Oops.
Something similar can happen with disk, but we are moving out of cpu-scheduler arena with that.
Quoted 7 lines
Quoted 2 lines
[I do not have SMP nearby to demonstrate it, anybody wanting to benchmark a bit?]
This is definitely the case but there is one huge difference. If you have 2x1Ghz non HT processors then the fastest a single threaded task can run is at 1Ghz. If you have 1x2Ghz HT processor the fastest a single threaded task can run is 2Ghz.
Well, gigaherz is not the *only* important thing.
On 2x1GHz, 2GB/sec RAM bandwidth, fastest a single threaded task can run is 1GHz, 2GB/sec. If you run two of them, it is 1GHz, *1*GB/sec. So you still have effect similar to hyperthreading. And yes, it can be measured.
stress runs two tasks walking over 10MB of memory, just for fun. Look:
[Lefik is dual-p3; according to you two mem stressers should run about same speed as one of them. That's not the case:]
machek@lefik:~/misc$ ./stress tenmega Process 1665 started at 1072522582. machek@lefik:~/misc$ Process 1665 done at 1072522695 (113 sec).
machek@lefik:~/misc$ ./stress tenmega tenmega Process 1669 started at 1072522722. Process 1670 started at 1072522722. machek@lefik:~/misc$ Process 1670 done at 1072522895 (173 sec). Process 1669 done at 1072522903 (181 sec).
machek@lefik:~/misc$
And yes, that machine does have two cpus:
machek@lefik:~/misc$ cat /proc/cpuinfo processor : 0 vendor_id : GenuineIntel cpu family : 6 model : 8 model name : Pentium III (Coppermine) stepping : 3 cpu MHz : 801.828 cache size : 256 KB physical id : 0 siblings : 1 fdiv_bug : no hlt_bug : no f00f_bug : no coma_bug : no fpu : yes fpu_exception : yes cpuid level : 2 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 mmx fxsr sse bogomips : 1599.07
processor : 1vendor_id : GenuineIntel
cpu family : 6
model : 8
model name : Pentium III (Coppermine)
stepping : 3
cpu MHz : 801.828
cache size : 256 KB
physical id : 0
siblings : 1fdiv_bug : nohlt_bug : nof00f_bug : nocoma_bug : no
fpu : yesfpu_exception : yes
cpuid level : 2
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge
mca cmov pat pse36 mmx fxsr sse
bogomips : 1602.35
machek@lefik:~/misc$
So... even on normal SMP, "task-on-other-cpu-slows-down-task-on-this-cpu" effect exists. Okay, it is not as visible as on HT machine (50% slowdown), but its definitely there.
Pavel
/* Copyright 1999-2003 Pavel Machek, distribute under GPLv2 */
#define MEM 20*1024
#define RAMSIZE (8*1024*1024)
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <sys/stat.h>
#include <sys/types.h>
#include <sys/fcntl.h>
#include <time.h>
voidmain( int argc, char *argv[] )
{
unsigned long i;
if (!argc)
{
printf( "stress loop|memread|tenmega|mem|write|eatmem ...\n" ); return;
}
for (i=0; i<argc; i++)
{
if (!strcmp( argv[i], "loop" ))
if (!fork())
while (1);
if (!strcmp( argv[i], "eatmem" ))
if (!fork())
while(1) {
char *c = malloc(4096);
if (c) *c='a';
}
if (!strcmp( argv[i], "memread" ))
if (!fork())
{
char *p = malloc( RAMSIZE );
for( i=0; i<RAMSIZE; i++ ) p[i]=1;
while( 1 )
{
int a;
for( i=0; i<RAMSIZE; i++ ) a+=p[i];
}
}
if (!strcmp( argv[i], "tenmega" ))
if (!fork())
{
char *p = malloc( 10*1024*1024 );
int i, j, start;
printf( "Process %d started at %d.\n", getpid(), start = time(NULL));
for( i=0; i<10*1024*1024; i++ ) p[i]=1;
for( j=1; j<1000; j++ )
{
volatile int a;
for( i=0; i<10*1024*1024; i++ ) a+=p[i];
}
printf( "Process %d done at %ld (%ld sec).\n", getpid(), (long) time(NULL), time(NULL)-start);
exit(0);
}
if (!strcmp( argv[i], "mem" ))
if (!fork())
{
char *p = malloc( RAMSIZE );
while( 1 )
{
for( i=0; i<RAMSIZE; i++ ) p[i]=1;
sleep( 60 );
}
}
if (!strcmp( argv[i], "write" ))
if (!fork())
{
char namebuf[1024];
int h;
char buf[1024]="Signature of something rather strange ;-)";
sprintf( namebuf, "/tmp/stresstest.delme.%d", getpid() );
h = creat( namebuf, 0666 );
if (h<0) { printf( "Creat failed: %m\n" ); exit(0); }
while( 1 )
for( i=0; i<MEM; i++ )
{
if (lseek( h, i*1024, SEEK_SET )<0) { printf( "Seek failed: %m\n" ); exit(0); }
if (write( h, buf, 1024 )<0) { printf( "Write failed: %m\n" ); exit(0); }
}
} }
}
Signature · 2 lines
When do you have a heart between your knees? [Johanka's followup: and *two* hearts?]
Quoted 4 lines · Pavel Machek
So... even on normal SMP, "task-on-other-cpu-slows-down-task-on-this-cpu" effect exists. Okay, it is not as visible as on HT machine (50% slowdown), but its definitely there.
Sure but I think we're getting pedantic here. The problem is really simple - a uniprocessor HT desktop booted in SMP mode feels half the speed while running setiathome (or video encoding or whatever cpu bound task) compared to booting it in UP mode. So, ironically, enabling the HT makes the machine feel slower when running multiple tasks. And there will be a heck of a lot of these in the future.
Con
Quoted 18 lines · Pavel Machek
Hi!
Quoted 6 lines
Quoted 6 lines
BTW this is going to be an issue even on normal (non-HT) systems. Imagine memory-bound scientific task on CPU0 and nice -20 memory-bound seti&home at CPU1. Even without hyperthreading, your scientific task is going to run at 50% of speed and seti&home is going to get second half. Oops.
Quoted 8 lines
Quoted 6 lines
Something similar can happen with disk, but we are moving out of cpu-scheduler arena with that.
[I do not have SMP nearby to demonstrate it, anybody wanting to benchmark a bit?]
This is definitely the case but there is one huge difference. If you have 2x1Ghz non HT processors then the fastest a single threaded task can run is at 1Ghz. If you have 1x2Ghz HT processor the fastest a single threaded task can run is 2Ghz.
Quoted 7 lines
Well, gigaherz is not the *only* important thing.
On 2x1GHz, 2GB/sec RAM bandwidth, fastest a single threaded task can run is 1GHz, 2GB/sec. If you run two of them, it is 1GHz, *1*GB/sec. So you still have effect similar to hyperthreading. And yes, it can be measured.
Hi Pavel,
Sure this might be a real problem sometimes, but I don't see the CPU scheduler ever handling it unless we want to add a few kitchen sinks to its nice lean code as well.
If the need really arises, then probably a userspace daemon could do it.
Hi!
Quoted 26 lines
Quoted 18 lines
Quoted 9 lines
Quoted 9 lines
BTW this is going to be an issue even on normal (non-HT) systems. Imagine memory-bound scientific task on CPU0 and nice -20 memory-bound seti&home at CPU1. Even without hyperthreading, your scientific task is going to run at 50% of speed and seti&home is going to get second half. Oops.
Something similar can happen with disk, but we are moving out of cpu-scheduler arena with that.
Quoted 8 lines
Quoted 3 lines
[I do not have SMP nearby to demonstrate it, anybody wanting to benchmark a bit?]
This is definitely the case but there is one huge difference. If you have 2x1Ghz non HT processors then the fastest a single threaded task can run is at 1Ghz. If you have 1x2Ghz HT processor the fastest a single threaded task can run is 2Ghz.
Quoted 7 lines
Well, gigaherz is not the *only* important thing.
On 2x1GHz, 2GB/sec RAM bandwidth, fastest a single threaded task can run is 1GHz, 2GB/sec. If you run two of them, it is 1GHz, *1*GB/sec. So you still have effect similar to hyperthreading. And yes, it can be measured.
Quoted 4 lines
Hi Pavel,
Sure this might be a real problem sometimes, but I don't see the CPU scheduler ever handling it unless we want to add a few kitchen sinks to its nice lean code as well.
Why is it a problem? If you are handling HT case, anyway, it should be fairly easy to say "imagine it is HT system, not SMP one", and poof, problem magically goes away.
Pavel
/*
* .----~~| * \ |
* ~~~~~~
*/
[Ready-made kitchen-sink for scheduler :-)))]
Signature · 2 lines
When do you have a heart between your knees? [Johanka's followup: and *two* hearts?]
In article <[email protected]>, Con Kolivas <[email protected]> wrote:
| On Sat, 27 Dec 2003 22:09, Pavel Machek wrote: | > So... even on normal SMP, | > "task-on-other-cpu-slows-down-task-on-this-cpu" effect exists. Okay, | > it is not as visible as on HT machine (50% slowdown), but its | > definitely there. | | Sure but I think we're getting pedantic here. The problem is really simple - a | uniprocessor HT desktop booted in SMP mode feels half the speed while running | setiathome (or video encoding or whatever cpu bound task) compared to booting | it in UP mode. So, ironically, enabling the HT makes the machine feel slower | when running multiple tasks. And there will be a heck of a lot of these in | the future.
Let me put forth a thought, without a solution. In the case you describe, what is needed, and not provided in hardware, is a way to do priority within the CPU, so in the case of a contested resource there is a way to ensure the process we wish wins.
Since that seems unavailable in Intel, do other CPUs do better (or different)?
Signature · 3 lines
bill davidsen <[email protected]> CTO, TMR Associates, Inc Doing interesting things with little computers since 1979.
In article <[email protected]>, =?ISO-8859-1?Q?Mika_Penttil=E4?= <[email protected]> wrote:
| heh...and the situation gets even worse when you add cpus, with 16way | you get only 1/16 of the speed ;)
No, not when you add CPUs, when you add siblings. There's a big difference, since sibs compete for cache on the chip, and some execution units (FPU?).
Signature · 3 lines
bill davidsen <[email protected]> CTO, TMR Associates, Inc Doing interesting things with little computers since 1979.
In article <[email protected]>, Con Kolivas <[email protected]> wrote:
| On Wed, 24 Dec 2003 02:51, bill davidsen wrote: | > There are two goals here. Not having a batch process on one siling makes | > sense, and I'm going to try Con's patch after I try Nick's latest. | > Actually, if they play nicely I would use both, batch would be very | > useful for nightly report generation on servers. | | No hope of them playing nicely, but at some later stage I might resync on top | of Nick's work if I like the direction it takes (which looks likely!) | | > But WRT the whole HT scheduling, it would seem that ideally you want to | > schedule the two (or N) processes which have the lowest aggregate cache | > thrash, if you had a way to determine that. I suspect that a process | > which had a small itterative inner loop with a code+data footprint of | > 2-3k would coexist well with almost anything else. Minimizing the FPU | > contention also would improve performance, no doubt. I don't know that | > there are the tools at the moment to get this information, but it seems | > as though until it's available any scheduling will be working in the | > dark to some extent. | | Impossible with current tools. Only userspace would have a chance of | predicting this and the simple rule we work off is that userspace can't be | trusted so this does not appear doable in the foreseeable future.
Glad you agree, but this makes improvement dificult.
Signature · 3 lines
bill davidsen <[email protected]> CTO, TMR Associates, Inc Doing interesting things with little computers since 1979.
Quoted 10 lines · Pavel Machek
Quoted 3 lines
Why do I need hyper-thread aware batch scheduling?
If you have a hyperthread (P4HT) processor and run it as two logical cpus you can have a very low priority task running that can consume 50% of your physical cpu's capacity no matter how high priority tasks you are running. For example if you use the distributed computing client setiathome you will be effectively be running at half your cpu's speed even if you run setiathome at nice 20. Batch scheduling for normal cpus allows only idle time to be used for batch tasks, and for HT cpus only allows idle time when both logical cpus are idle.
BTW this is going to be an issue even on normal (non-HT) systems. Imagine memory-bound scientific task on CPU0 and nice -20 memory-bound seti&home at CPU1. Even without hyperthreading, your scientific task is going to run at 50% of speed and seti&home is going to get second half. Oops.
Yes and even worse, if you stop running setiathome the scientific task *still* only gets half the available CPU!
The difference is that with HT running a task on one sibling actually does (or can) slow the other. That's not true with true SMP, at least not directly, since the resourses shared (memory and disk) are much farther away from the CPU.
Signature · 3 lines
bill davidsen <[email protected]> CTO TMR Associates, Inc Doing interesting things with small computers since 1979
Quoted 1 line · Con Kolivas
I discussed this with Ingo and that's the sort of thing we thought of. Perhaps a relative crossover of 10 dynamic priorities and an absolute crossover of 5 static priorities before things got queued together. This is really only required for the UP HT case.
What? Do siblings in Xeons not compete for cache and memory bandwidth, executions units, and the like?
Signature · 3 lines
bill davidsen <[email protected]> CTO TMR Associates, Inc Doing interesting things with small computers since 1979
Quoted 2 lines · Bill Davidsen
Yes and even worse, if you stop running setiathome the scientific task *still* only gets half the available CPU!
Look that this is not true. If one core is not running any task, the idle task (if not polling) does "hlt" and the "what they call Fetch And Deliver" engine will be dedicated to the other core. Also, because the halted core not not issue any op to the execution engine, full resources will be available for the running task. There are many docs available inside the Intel developer web site that explain this.
- Davide
Quoted 7 lines · Davide Libenzi
Quoted 2 lines · Bill Davidsen
Yes and even worse, if you stop running setiathome the scientific task *still* only gets half the available CPU!
Look that this is not true. If one core is not running any task, the idle task (if not polling) does "hlt" and the "what they call Fetch And
What Bill said was:
Quoted 6 lines
Quoted 3 lines
memory-bound seti&home at CPU1. Even without hyperthreading, your scientific task is going to run at 50% of speed and seti&home is going to get second half. Oops.
Yes and even worse, if you stop running setiathome the scientific task *still* only gets half the available CPU!
So Bill is pointing out that on a *normal* SMP, you get 50% whether or not the other processor is busy.
Quoted 4 lines
The difference is that with HT running a task on one sibling actually does (or can) slow the other. That's not true with true SMP, at least not directly, since the resourses shared (memory and disk) are much farther away from the CPU.
And this is where Bill talks about issues like the one you mentioned about sharing the dispatch engine.
So I think you and Bill are actually saying the same exact thing.
Quoted 15 lines · On Fri, 2 Jan 2004 [email protected] wrote:
Quoted 7 lines · Davide Libenzi
Quoted 2 lines · Bill Davidsen
Yes and even worse, if you stop running setiathome the scientific task *still* only gets half the available CPU!
Look that this is not true. If one core is not running any task, the idle task (if not polling) does "hlt" and the "what they call Fetch And
What Bill said was:
Quoted 3 lines
Quoted 3 lines
memory-bound seti&home at CPU1. Even without hyperthreading, your scientific task is going to run at 50% of speed and seti&home is going to get second half. Oops.
Quoted 5 lines
Quoted 2 lines
Yes and even worse, if you stop running setiathome the scientific task *still* only gets half the available CPU!
So Bill is pointing out that on a *normal* SMP, you get 50% whether or not the other processor is busy.
Define 50% ;) In case you are talking about 50% of all the available resources, in case of a 2 way SMP this is pretty obvious since a single thread cannot run simultaneuosly on two CPUs. In case of an HT core, this is not true since the single thread will expand using the whole core resources. Note though that it won't be an expansion to 100%, and this is the reason of the existence of the HT technology. That is, even with monster length pipelines, the CPU is not able to keep exec units 100% full using a single dispatch unit.
- Davide