dynamic sched timeslices
Subsystems (2)
Hi,
attached patch allows userspace to tune the scheduling timeslices. It can be used for a couple of things:
* Tune a workload for batch processing:
You'd probably wnat to use long timeslices in order to not reschedule as often to make good use of your CPU caches
* Tune a workload for interactive use:
Under load, you may want to reduce the scedulilng latencies by using shorter timeslices (and there are situations where the interactiviy tweak -- even if they were perfect -- can't save you).
* Tune the ration betweeen maximum and minimum timeslices to make nice much nicer e.g.
The patch exports /proc/sys/kernel/max_timeslice and min_timeslice, unites are us. It also exports HZ (readonly).
The patch implementes the desktop boot parameter which introduces shorter timeslices.
Patch is from andrea and is in our 2.4 tree; 2.6 port was done by me and straightforward.
Regards,
Signature · 2 lines
Kurt Garloff <[email protected]> Cologne, DE SUSE LINUX AG, Nuernberg, DE SUSE Labs (Head)
include/linux/sched.h +1 -0
@@ -176,6 +176,7 @@ extern unsigned long cache_decay_ticks;#define MAX_SCHEDULE_TIMEOUT LONG_MAXextern signed long FASTCALL (schedule_timeout (signed long timeout)); asmlinkage void schedule(void); +extern int max_timeslice, min_timeslice;struct namespace ;
include/linux/sysctl.h +3 -0
@@ -134,6 +134,9 @@ enumKERN_KDB =64 ,/* int: kdb on/off */ KERN_S390_HZ_TIMER =65 ,/* int: hz timer on or off */ KERN_DUMP =66 ,/* directory: dump parameters */ +KERN_MAXTIMESLICE =67 ,/* int: nice -20 max timeslice */ +KERN_MINTIMESLICE =68 ,/* int: nice +19 min timeslice */ + KERN_HZ=69, /* unsigned long: interal kernel HZ */ };
kernel/sched.c +23 -6
@@ -79,12 +79,19 @@ EXPORT_SYMBOL(dump_oncpu);/* * These are the 'tuning knobs' of the scheduler: * -* Minimum timeslice is 10 msecs, default timeslice is 100 msecs, -* maximum timeslice is 200 msecs. Timeslices get refilled after -* they expire. -*/ -#define MIN_TIMESLICE ( 10 * HZ / 1000) -#define MAX_TIMESLICE (200 * HZ / 1000) +* Minimum timeslice is 10 msecs, default timeslice is 150 msecs, +* maximum timeslice is 300 msecs. Timeslices get refilled after +* they expire. +*/ +#define __MIN_TIMESLICE 10000 +#define __MAX_TIMESLICE 300000 +#define __MIN_TIMESLICE_DESKTOP 2000 +#define __MAX_TIMESLICE_DESKTOP 60000 +/* the sysctl values are exported in usec units to userspace */ +int max_timeslice = __MAX_TIMESLICE, min_timeslice = __MIN_TIMESLICE; +#define MAX_TIMESLICE ((max_timeslice * HZ + 999999) / 1000000) +#define MIN_TIMESLICE ((min_timeslice * HZ + 999999) / 1000000) +#define ON_RUNQUEUE_WEIGHT 30#define CHILD_PENALTY 95#define PARENT_PENALTY 100
@@ -2965,3 +2972,13 @@ task_t *kdb_cpu_curr(int cpu) return(cpu_curr(cpu)); }#endif + +static int __init init_desktop (char *str) +{ + min_timeslice = __MIN_TIMESLICE_DESKTOP; + max_timeslice = __MAX_TIMESLICE_DESKTOP; +return 1 ; +} +__setup ("desktop" , init_desktop); + +
kernel/sysctl.c +25 -0
@@ -167,6 +167,7 @@ static void register_proc_table(ctl_tablstatic void unregister_proc_table (ctl_table *,struct proc_dir_entry *);#endif +static unsigned long __HZ =HZ ;/* The default sysctl tables: */ static ctl_table root_table[] = {
@@ -642,6 +643,30 @@ static ctl_table kern_table[] = { .mode = 0444, .proc_handler = &proc_dointvec, }, + { + .ctl_name = KERN_MAXTIMESLICE, + .procname = "max-timeslice", + .data = &max_timeslice, + .maxlen = sizeof(int), + .mode = 0644, + .proc_handler = &proc_dointvec, + }, + { + .ctl_name = KERN_MINTIMESLICE, + .procname = "min-timeslice", + .data = &min_timeslice, + .maxlen = sizeof(int), + .mode = 0644, + .proc_handler = &proc_dointvec, + }, + { + .ctl_name = KERN_HZ, + .procname = "HZ", + .data = &__HZ, + .maxlen = sizeof(long), + .mode = 0444, + .proc_handler = &proc_dointvec, + }, { .ctl_name = 0 } };
Quoted 14 lines · Kurt Garloff
Hi,
attached patch allows userspace to tune the scheduling timeslices. It can be used for a couple of things:
* Tune a workload for batch processing:
You'd probably wnat to use long timeslices in order to not reschedule as often to make good use of your CPU caches
* Tune a workload for interactive use:
Under load, you may want to reduce the scedulilng latencies by using shorter timeslices (and there are situations where the interactiviy tweak -- even if they were perfect -- can't save you).
* Tune the ration betweeen maximum and minimum timeslices to make nice much nicer e.g.
Quoted 7 lines
The patch exports
/proc/sys/kernel/max_timesliceandmin_timeslice, unites are us. It also exports HZ (readonly).
The patch implementes the desktop boot parameter which introduces shorter timeslices.Patch is from andrea and is in our 2.4 tree; 2.6 port was done by me and straightforward.
Remove the silly desktop boot parameter and the patch looks basically okay to me.
I remember we had a more complete patch to allow tuning the scheduler through sysctls in -mm once, though. Questions is why that one wasn't merged and if the same reasons apply to a 'light' version.
Hi Christoph,
quick response!
Quoted 2 lines · Christoph Hellwig
Remove the silly desktop boot parameter and the patch looks basically okay to me.
See attachment.
Quoted 3 lines
I remember we had a more complete patch to allow tuning the scheduler through sysctls in -mm once, though. Questions is why that one wasn't merged and if the same reasons apply to a 'light' version.
Hmm, I fail to remember unfortunately. Probably it had too many knobs. Andrew?
Regards,
Signature · 2 lines
Kurt Garloff <[email protected]> Cologne, DE SUSE LINUX AG, Nuernberg, DE SUSE Labs (Head)
include/linux/sched.h +1 -0
@@ -176,6 +176,7 @@#define MAX_SCHEDULE_TIMEOUT LONG_MAXextern signed long FASTCALL (schedule_timeout (signed long timeout)); asmlinkage void schedule(void); +extern int max_timeslice, min_timeslice;struct namespace ;
include/linux/sysctl.h +3 -0
@@ -134,6 +134,9 @@KERN_KDB =64 ,/* int: kdb on/off */ KERN_S390_HZ_TIMER =65 ,/* int: hz timer on or off */ KERN_DUMP =66 ,/* directory: dump parameters */ +KERN_MAXTIMESLICE =67 ,/* int: nice -20 max timeslice */ +KERN_MINTIMESLICE =68 ,/* int: nice +19 min timeslice */ + KERN_HZ=69, /* unsigned long: internal kernel HZ */ };
kernel/sched.c +11 -6
@@ -79,12 +79,17 @@/* * These are the 'tuning knobs' of the scheduler: * -* Minimum timeslice is 10 msecs, default timeslice is 100 msecs, -* maximum timeslice is 200 msecs. Timeslices get refilled after -* they expire. -*/ -#define MIN_TIMESLICE ( 10 * HZ / 1000) -#define MAX_TIMESLICE (200 * HZ / 1000) +* Minimum timeslice is 10 msecs, default timeslice is 150 msecs, +* maximum timeslice is 300 msecs. Timeslices get refilled after +* they expire. +*/ +#define __MIN_TIMESLICE 10000 +#define __MAX_TIMESLICE 300000 +/* the sysctl values are exported in usec units to userspace */ +int max_timeslice = __MAX_TIMESLICE, min_timeslice = __MIN_TIMESLICE; +#define MAX_TIMESLICE ((max_timeslice * HZ + 999999) / 1000000) +#define MIN_TIMESLICE ((min_timeslice * HZ + 999999) / 1000000) +#define ON_RUNQUEUE_WEIGHT 30#define CHILD_PENALTY 95#define PARENT_PENALTY 100
kernel/sysctl.c +25 -0
@@ -167,6 +167,7 @@static void unregister_proc_table (ctl_table *,struct proc_dir_entry *);#endif +static unsigned long __HZ =HZ ;/* The default sysctl tables: */ static ctl_table root_table[] = {
@@ -642,6 +643,30 @@ .mode = 0444, .proc_handler = &proc_dointvec, }, + { + .ctl_name = KERN_MAXTIMESLICE, + .procname = "max-timeslice", + .data = &max_timeslice, + .maxlen = sizeof(int), + .mode = 0644, + .proc_handler = &proc_dointvec, + }, + { + .ctl_name = KERN_MINTIMESLICE, + .procname = "min-timeslice", + .data = &min_timeslice, + .maxlen = sizeof(int), + .mode = 0644, + .proc_handler = &proc_dointvec, + }, + { + .ctl_name = KERN_HZ, + .procname = "HZ", + .data = &__HZ, + .maxlen = sizeof(long), + .mode = 0444, + .proc_handler = &proc_dointvec, + }, { .ctl_name = 0 } };
Quoted 7 lines · Kurt Garloff
Quoted 4 lines
I remember we had a more complete patch to allow tuning the scheduler through sysctls in -mm once, though. Questions is why that one wasn't merged and if the same reasons apply to a 'light' version.
Hmm, I fail to remember unfortunately. Probably it had too many knobs. Andrew?
It had a zillion knobs, and was mainly for developers.
Your patch didn't come with any subjective or measured testing results. In theory, the scheduler should magically tune itself to the current workload. If your patch is indeed necessary then this may point at a bug in the current CPU scheduler.
Please tell us more...
Hi Andrew,
Quoted 2 lines · Andrew Morton
Kurt Garloff <[email protected]> wrote:
Your patch didn't come with any subjective or measured testing results.
We've done some measurements with 2.4 and O(1):
* HZ=1000 cost about 1.5% perf. on a kernel compile (plus problems with lost timer ticks)
* Seting the scheduling timeslices from 1ms--30ms rather than 10ms thr 300ms cost another ~3% kernel compile performance
* Depending on the workload the effect can be larger. Numbercrunching would come to mind.
* Some people are unhappy that nice is not nice enough.
Quoted 2 lines
In
theory, the scheduler should magically tune itself to the current workload.
No, the computer can not take the decision whether a machine is sitting in a machine room and mainly doing batch processing or whether it's used by somebody sitting in front of it as workstation.
You can add heuristics and look at the load and sleep_avg of processes, and scale timeslices dynamically, but these heuristics are IMVHO a very bad idea. They tend to break in subtle ways and disallow to reproduce benchmark numbers etc.
I do know that the priorites do ensure that interactive processes have some bonus, so even with long timeslices a system should be usable. But heuristics fail and some situtaions with high load just can't be solved by such bonuses.
Quoted 2 lines
If your patch is indeed necessary then this may point at a bug in the current CPU scheduler.
No, why should it? How should the computer know what the user wants if he has no way to tell him?
It's a classical throughput vs. latency tradeoff and the patch allows the user to set it. I'm sure some people are willing to have long timeslices in order to gain 5% and don't care about the sched latencies.
Regards,
Signature · 2 lines
Kurt Garloff <[email protected]> Cologne, DE SUSE LINUX AG, Nuernberg, DE SUSE Labs (Head)
Quoted 10 lines · Kurt Garloff
Quoted 1 line · Andrew Morton
Your patch didn't come with any subjective or measured testing results.
We've done some measurements with 2.4 and O(1):
* HZ=1000 cost about 1.5% perf. on a kernel compile (plus problems with lost timer ticks)
* Seting the scheduling timeslices from 1ms--30ms rather than 10ms thr 300ms cost another ~3% kernel compile performance
* Depending on the workload the effect can be larger. Numbercrunching would come to mind.
2.4 O(1) effects do not directly apply with 2.6
Dropping Hz will save you performance for sure on 2.6.
Changing the timeslices in 2.6 will be disappointing, though. Although the apparent timeslice of nice 0 tasks is 102ms, interactive tasks round robin at 10ms. If you drop the timeslice to 10ms you will not improve the interactive feel but you will speed up expiration instead which will almost certainly worsen interactive feel. If you drop timeslices below 10ms you will get significant cache trashing and drop in performance (which your 2.4 results confirm).
Increasing timeslices does benefit pure number crunching workloads. The benchmarking I've done using cache intensive workloads (which are the most likely to benefit) show you are chasing diminishing returns, though. You can mathematically model them based on the fact that keeping a task bound to a cpu instead of shifting it to another cpu on SMP saves about 2ms processing time on P4. Suffice to say the benefit is only worth it if you do nothing but cpu intensive things, and becomes virtually insignificant beyond 200ms. On other architecture with longer cache decays you will benefit more; arch/i386/mach-voyager seems the longest at 20ms.
Quoted 3 lines
It's a classical throughput vs. latency tradeoff and the patch allows the user to set it. I'm sure some people are willing to have long timeslices in order to gain 5% and don't care about the sched latencies.
As you can see from the above it is not that clear cut.
Con
Hi Con,
Quoted 9 lines · Con Kolivas
2.4 O(1) effects do not directly apply with 2.6
Dropping Hz will save you performance for sure on 2.6.
Changing the timeslices in 2.6 will be disappointing, though. Although the apparent timeslice of nice 0 tasks is 102ms, interactive tasks round robin at 10ms. If you drop the timeslice to 10ms you will not improve the interactive feel but you will speed up expiration instead which will almost certainly worsen interactive feel.
If you have a system with an easy workload (say one clear CPU hog and one interactive job), things are easy. The fact that you preempt the not-yet expired CPU hog is enough.
That's easy, and that worked with 2.4 O(1) (if tweaked a bit to estimate interactiveness better, see other patch) and it works with 2.6.
Things start to get difficult if you have something like a calculation program with a non-multithreaded GUI. It will look like a CPU hog and still you'd like to see it responsive. Now add a second CPU hog.
The kernel can not fix this problem, but it can limit the damage by not having too long timeslices.
There are other scenarios where the preemption will not solve all problems.
Think two interactive processes, one playing audio, another one being your shell. The audio player may take the CPU for extended periods of times occasionally to decode the next N ogg frames. You still want the shell to react promptly, but it can't ... Thus you wish the timeslice not being too long.
Thus you'll set them not too long for desktop kind of machines to not have to rely completely on the interactiveness estimator.
Quoted 3 lines
If you drop timeslices below 10ms you will get
significant cache trashing and drop in performance (which your 2.4 results confirm).
No doubt. Don't overdo it. It's a tradeoff. If you impact throughput too much, you'll not enjoy the short latency ;-)
Quoted 9 lines
Increasing timeslices does benefit pure number crunching workloads. The benchmarking I've done using cache intensive workloads (which are the most likely to benefit) show you are chasing diminishing returns, though. You can mathematically model them based on the fact that keeping a task bound to a cpu instead of shifting it to another cpu on SMP saves about 2ms processing time on P4. Suffice to say the benefit is only worth it if you do nothing but cpu intensive things, and becomes virtually insignificant beyond 200ms. On other architecture with longer cache decays you will benefit more;
arch/i386/mach-voyagerseems the longest at 20ms.
That's why I think we should offer the tunables.
Regards,
Signature · 3 lines
Kurt Garloff <[email protected]> [Koeln, DE] Physics:Plasma modeling <[email protected]> [TU Eindhoven, NL] Linux: SUSE Labs (Head) <[email protected]> [SUSE Nuernberg, DE]
Quoted 34 lines · Kurt Garloff
Hi Con,
Quoted 9 lines · Con Kolivas
2.4 O(1) effects do not directly apply with 2.6
Dropping Hz will save you performance for sure on 2.6.
Changing the timeslices in 2.6 will be disappointing, though. Although the apparent timeslice of nice 0 tasks is 102ms, interactive tasks round robin at 10ms. If you drop the timeslice to 10ms you will not improve the interactive feel but you will speed up expiration instead which will almost certainly worsen interactive feel.
If you have a system with an easy workload (say one clear CPU hog and one interactive job), things are easy. The fact that you preempt the not-yet expired CPU hog is enough.
That's easy, and that worked with 2.4 O(1) (if tweaked a bit to estimate interactiveness better, see other patch) and it works with 2.6.Things start to get difficult if you have something like a calculation program with a non-multithreaded GUI. It will look like a CPU hog and still you'd like to see it responsive. Now add a second CPU hog.
The kernel can not fix this problem, but it can limit the damage by not having too long timeslices.
There are other scenarios where the preemption will not solve all problems.
Think two interactive processes, one playing audio, another one being your shell. The audio player may take the CPU for extended periods of times occasionally to decode the next N ogg frames. You still want the shell to react promptly, but it can't ... Thus you wish the timeslice not being too long.
Quoted 2 lines
Thus you'll set them not too long for desktop kind of machines to not have to rely completely on the interactiveness estimator.
I'm not arguing with your logic I'm saying you will ruin the estimator in the process because it isn't just about timeslices and preemption. Ultimately you're still only give me theoretical cases which are exactly what were my test cases during development. Find a test case that you can prove it does something rather than a theoretical one. Oh and a test case should be one where it is the cpu scheduler that is responsible, not one where you're fixing problems due to poor hardware config with alsa, oss, IDE, dma settings etc.
Quoted 18 lines
Quoted 3 lines
If you drop timeslices below 10ms you will get significant cache trashing and drop in performance (which your 2.4 results confirm).
No doubt. Don't overdo it. It's a tradeoff. If you impact throughput too much, you'll not enjoy the short latency ;-)
Quoted 10 lines
Increasing timeslices does benefit pure number crunching workloads. The benchmarking I've done using cache intensive workloads (which are the most likely to benefit) show you are chasing diminishing returns, though. You can mathematically model them based on the fact that keeping a task bound to a cpu instead of shifting it to another cpu on SMP saves about 2ms processing time on P4. Suffice to say the benefit is only worth it if you do nothing but cpu intensive things, and becomes virtually insignificant beyond 200ms. On other architecture with longer cache decays you will benefit more;
arch/i386/mach-voyagerseems the longest at 20ms.
Quoted 1 line
That's why I think we should offer the tunables.
If your workload is so dedicated to just number crunching it isn't hard to add a zero to maximum timeslice in kernel/sced.c. Then again this is just semantics about how to tune it so I don't really care, but I'm sure the maintainer wants proof that changing it shows some real world improvement.
Con
Hi Con,
Quoted 4 lines · Con Kolivas
Quoted 1 line
That's why I think we should offer the tunables.
If your workload is so dedicated to just number crunching it isn't hard to add a zero to maximum timeslice in
kernel/sced.c.
Of course I can compile a custom kernel for myself and tune all sorts of things. But this is not the way most Linux users want to use Linux any more. Actually that's a long time ago.
Best regards,
Signature · 3 lines
Kurt Garloff <[email protected]> [Koeln, DE] Physics:Plasma modeling <[email protected]> [TU Eindhoven, NL] Linux: SUSE Labs (Head) <[email protected]> [SUSE Nuernberg, DE]
Quoted 4 lines · Kurt Garloff
Hi Con,
On Wed, Mar 17, 2004 at 07:45:02AM +1100, Con Kolivas wrote:
Quoted 9 lines
Quoted 4 lines
Quoted 1 line
That's why I think we should offer the tunables.
If your workload is so dedicated to just number crunching it isn't hard to add a zero to maximum timeslice in
kernel/sced.c.Of course I can compile a custom kernel for myself and tune all sorts of things. But this is not the way most Linux users want to use Linux any more. Actually that's a long time ago.
I don't think we should be averse to offering a couple of nice high-level scheduler tunables. But I do think we should have testing results which clearly show that they provide some benefit, and we should agree that the scheduler cannot provide the same benefit automagically.
Apologies in advance if we've seen those testing results and I missed it.
Quoting Andrew Morton <[email protected]>:
Quoted 16 lines
Quoted 7 lines · Kurt Garloff
Hi Con,
Quoted 3 lines · Con Kolivas
Quoted 1 line
That's why I think we should offer the tunables.
If your workload is so dedicated to just number crunching it isn't hard
to add
Quoted 6 lines
Quoted 1 line
a zero to maximum timeslice in
kernel/sced.c.Of course I can compile a custom kernel for myself and tune all sorts of things. But this is not the way most Linux users want to use Linux any more. Actually that's a long time ago.
Quoted 6 lines
I don't think we should be averse to offering a couple of nice high-level scheduler tunables. But I do think we should have testing results which clearly show that they provide some benefit, and we should agree that the scheduler cannot provide the same benefit automagically.
Apologies in advance if we've seen those testing results and I missed it.
Well that reply takes my message out of context. I'm not averse to tunables - if they do something.
The only evidence Kurt has shown so far is that he can decrease throughput. The rest is theoretical based on a scheduler that isn't the 2.6 kernel.
Con
Quoted 14 lines · Kurt Garloff
Hi,
attached patch allows userspace to tune the scheduling timeslices. It can be used for a couple of things:
* Tune a workload for batch processing:
You'd probably wnat to use long timeslices in order to not reschedule as often to make good use of your CPU caches
* Tune a workload for interactive use:
Under load, you may want to reduce the scedulilng latencies by using shorter timeslices (and there are situations where the interactiviy tweak -- even if they were perfect -- can't save you).
* Tune the ration betweeen maximum and minimum timeslices to make nice much nicer e.g.
Quoted 8 lines
The patch exports
/proc/sys/kernel/max_timesliceandmin_timeslice, unites are us. It also exports HZ (readonly).
The patch implementes the desktop boot parameter which introduces shorter timeslices.Patch is from andrea and is in our 2.4 tree; 2.6 port was done by me and straightforward.
Regards,
If this doesn't change the total amount of CPU a process can get but lets a process tweak how its CPU time is divided up, then it sounds wonderful.
Hi Timothy,
Quoted 3 lines · Timothy Miller
If this doesn't change the total amount of CPU a process can get but lets a process tweak how its CPU time is divided up, then it sounds wonderful.
It let's the sysadming tune how often the kernel switches between processes competing for the CPU.
Regards,
Signature · 2 lines
Kurt Garloff <[email protected]> Cologne, DE SUSE LINUX AG, Nuernberg, DE SUSE Labs (Head)
Hi!
Quoted 11 lines
attached patch allows userspace to tune the scheduling timeslices. It can be used for a couple of things:
* Tune a workload for batch processing:
You'd probably wnat to use long timeslices in order to not reschedule as often to make good use of your CPU caches
* Tune a workload for interactive use:
Under load, you may want to reduce the scedulilng latencies by using shorter timeslices (and there are situations where the interactiviy tweak -- even if they were perfect -- can't save you).
* Tune the ration betweeen maximum and minimum timeslices to make nice much nicer e.g.
If you make ration much bigger, you are going to see priority inversion issues. Some kind of "boost priority when in kernel" would be needed...
Signature · 1 line
64 bytes from 195.113.31.123: icmp_seq=28 ttl=51 time=448769.1 ms