Saving power by cpu evacuation sched_max_capacity_pct=n
v2 cover Saving power by cpu evacuation sched_max_capacity_pct=n
Hi,
The idea of extending sched_mc_powersavings tunable for cpu evacuation was discussed at http://lwn.net/Articles/330309/
The summary of the discussion is as follows:
* Using sched_mc=3,4,5 to evacuate 1,2,4 cores is completely non-intuitive and broken interface. Ingo wanted to see if we can model a global percentile tunable that would map to core throttling.
* Peter Zijlstra wanted more justifications for throttling at the core level. Throttling may be a resource management problem rather than scheduler/load balancer
* CPU hotplug and cpuset/cgroup based cpu throttling are viable alternatives to this approach.
Changes in v2:
* Created a percentage knob sched_max_capacity_pct=n Defaults to 100, can be set to 75 or 50 to evacuate cores
* This patch is still a hack for discussion and has many limitations.
v1: http://lkml.org/lkml/2009/4/26/202
Into and parts from previous post for quick reference:
------------------------------------------------------
Objective:
----------
* Framework to evacuate tasks from cpus in order to force the cpu cores to stay at idle. Forcefully idling cores and packages can reduce power consumption.
* Fast response time and low OS overhead to moved tasks away from selected cpu packages. CPU hotplug is too heavyweight for this purpose
Use cases:
---------
* Ability to throttle the number of cores used in the system along with other power saving controls like cpufreq governors can enable the system to operate at a more power efficient operating point and still meet the design objectives.
* Facilitate thermal management by evacuating cores from hot cpu packages
Alternatives:
-------------
* CPU hotplug: Heavy weight and slow. Setting up and tear down of data structures involved. May need new fast or light weight notifications
* CPUSets: Exclusive CPU sets and partitioned sched domains involve rebuilding sched domains and relatively heavy weight for the purpose
The following patch is against 2.6.30-rc5 and will work only in an under utilised system (No of tasks <= number of cores).
Test results for ebizzy 8 threads at various sched_max_capacity_pct settings. The test platform is dual socket quad core x86 system (pre-Nehalem).
This is an interesting characteristics of the ebizzy benchmark where the following command line improved in performance as we evacuated cores! Perhaps cross-cache traffic... I will verify that next time.
ebizzy -s 4096 -t 8 -S 30
sched_mc_power_savings was set to 2 in the experiment
-----------------------------------------------------------------sched_max_capacity_pct No Cores Performance AvgPower
used Records/sec (Watts)
-----------------------------------------------------------------
100 8 1.00x 1.00y
87 7 1.03x 0.98y
75 6 1.06x 0.95y
62 5 1.26x 0.91y
50 4 1.15x 0.86y
-----------------------------------------------------------------
There were wide run variation with ebizzy. The purpose of the above data is to justify use of core evacuation for power vs performance trade-offs. The patch does not yet work for kernbench and other complex workloads/benchmarks. I even tried SPECjbb and did not get the expected CPU utilisation at various settings to reduce power consumption. The utilisation/power was much lower than expected.
ToDo:
-----
* Identify good benchmark to demonstrate benefits of cpu evacuation
* Make the core evacuation predictable under different system load conditions and workload characteristics. This is turning out to be a major challenge in this approach.
* Enhance framework to control which particular packages/cores will be evacuated, this is needed for thermal management. The CPU hotplug/cpuset approach will solve this problem.
I can experiment with different benchmarks/platforms and post results while the framework is being discussed.
Please let me know you comments and suggestions.
Thanks,
Vaidy
---
Vaidyanathan Srinivasan (2):
sched: loadbalancer hacks for forced packing of tasks
sched: add sched_max_capacity_pct
kernel/sched.c | 65 +++++++++++++++++++++++++++++++++++++++++++++++++++++++- 1 files changed, 64 insertions(+), 1 deletions(-)
RFC PATCH v2 1/2 sched: add sched_max_capacity_pct
Add a new sysfs variable that can be used by user space to pass the number of core to evacuate or force idle.
/sys/devices/system/cpu/sched_max_capacity_pct defaults to 100
This is percentage value that can be used to force idle cores. The percentage number shall be in steps corresponding to number of cores in the system.
On a 8 core system (dual socket quad core), each core step will be 12.5% rounded to 12%.
Echoing 88 will use 7 cores in the system:
% No of cores
100 8
87 7
75 6
62 5
50 4
...
...
This patch will evacuate only one package (50%) in ths case.
** This is a RFC patch for discussion ***
---
kernel/sched.c | 37 +++++++++++++++++++++++++++++++++++++ 1 files changed, 37 insertions(+), 0 deletions(-)
kernel/sched.c +37 -0
@@ -3291,6 +3291,9 @@ static inline int get_sd_load_idx(struct sched_domain *sd,#if defined(CONFIG_SCHED_MC ) || defined(CONFIG_SCHED_SMT ) + +int sched_evacuate_cores;/* No of forced-idle cores */ +/** * init_sd_power_savings_stats - Initialize power savings statistics for * the given sched_domain, during load balancing.
@@ -8604,6 +8607,37 @@ static ssize_t sched_mc_power_savings_store(struct sysdev_class *class, static SYSDEV_CLASS_ATTR(sched_mc_power_savings, 0644, sched_mc_power_savings_show, sched_mc_power_savings_store); + +static ssize_t sched_max_capacity_pct_show (struct sysdev_class *class, +char *page) +{ +int capacity; +/* Convert no of cores to system capacity percentage */ +/* FIXME: Will work only for non-threaded systems */ + capacity =100 - sched_evacuate_cores *100 / nr_cpu_ids; +return sprintf (page,"%u\n" , capacity); +} +static ssize_t sched_max_capacity_pct_store (struct sysdev_class *class, +const char *buf,size_t count) +{ +int capacity; +if (!sscanf (buf,"%u" , &capacity)) +return -EINVAL ; + +if (capacity <1 || capacity >100 ) +return -EINVAL ; + +/* Convert user provided percentage into no-of-cores to evacuate */ +/* FIXME: Will work only for non-threaded systems */ + sched_evacuate_cores = (101 - capacity) * nr_cpu_ids /100 ; +return count; +} + + +static SYSDEV_CLASS_ATTR(sched_max_capacity_pct, 0644, + sched_max_capacity_pct_show, + sched_max_capacity_pct_store); +#endif #ifdef CONFIG_SCHED_SMT
@@ -8635,6 +8669,9 @@ int __init sched_create_sysfs_power_savings_entries(struct sysdev_class *cls)if (!err &&mc_capable ()) err =sysfs_create_file (&cls->kset.kobj, &attr_sched_mc_power_savings.attr); +if (!err) + err =sysfs_create_file (&cls->kset.kobj, + &attr_sched_max_capacity_pct.attr);#endif return err; }
RFC PATCH v2 2/2 sched: loadbalancer hacks for forced packing of tasks
In reply to Vaidyanathan SrinivasanPack more tasks in a group so as to reduce number of CPUs used to run the work in the system.
Just for load balancing purpose, assume the group capacity has been increased by group_capacity_bump()
Hacks:
o Make non-idle cpus also perform powersave balance so that we can pull more tasks into the group
o Increase group capacity for calculation
o Increase load-balancing threshold so that even if a group is overloaded by group_capacity_bump(), consider it balanced
Basically if we want to evacuate 2 cores, the group capacity is increased by 2 (*SCHED_LOAD_SCALE) and the power save balancer will accommodate the tasks after selecting the group leader.
This will not work if the system is overloaded. Even after pulling 2 extra tasks, there could be tasks to fill the other package. At this point we are not yet reducing the group capacity of the other group.
*** RFC patch for discussion ***
---
kernel/sched.c | 28 +++++++++++++++++++++++++++- 1 files changed, 27 insertions(+), 1 deletions(-)
kernel/sched.c +27 -1
@@ -3234,6 +3234,7 @@ struct sd_lb_stats {int group_imb;/* Is there imbalance in this sd */ #if defined(CONFIG_SCHED_MC ) || defined(CONFIG_SCHED_SMT )int power_savings_balance;/* Is powersave balance needed for this sd */ +unsigned int group_capacity_bump;/* % increase in group capacity */ struct sched_group *group_min;/* Least loaded group in sd */ struct sched_group *group_leader;/* Group which relieves group_min */ unsigned long min_load_per_task;/* load_per_task in group_min */
@@ -3294,6 +3295,15 @@ static inline int get_sd_load_idx(struct sched_domain *sd,int sched_evacuate_cores;/* No of forced-idle cores */ +static inline unsigned int group_capacity_bump (struct sched_domain *sd) +{ + +if (sd->flags &SD_POWERSAVINGS_BALANCE ) +return sched_evacuate_cores; + +return 0 ; +} +/** * init_sd_power_savings_stats - Initialize power savings statistics for * the given sched_domain, during load balancing.
@@ -3309,12 +3319,14 @@ static inline void init_sd_power_savings_stats(struct sched_domain *sd,* Busy processors will not participate in power savings * balance. */ -if (idle ==CPU_NOT_IDLE || !(sd->flags &SD_POWERSAVINGS_BALANCE )) +if ((idle ==CPU_NOT_IDLE && !sched_evacuate_cores) || + !(sd->flags &SD_POWERSAVINGS_BALANCE )) sds->power_savings_balance =0 ;else { sds->power_savings_balance =1 ; sds->min_nr_running =ULONG_MAX ; sds->leader_nr_running =0 ; + sds->group_capacity_bump =group_capacity_bump (sd); } }
@@ -3436,6 +3448,12 @@ static inline int check_power_save_busiest_group(struct sd_lb_stats *sds, {return 0 ; } + +static inline unsigned int group_capacity_bump (struct sched_domain *sd) +{ +return 0 ; +} +#endif /* CONFIG_SCHED_MC || CONFIG_SCHED_SMT */
@@ -3568,6 +3586,8 @@ static inline void update_sd_lb_stats(struct sched_domain *sd, int this_cpu,if (local_group && balance && !(*balance))return ; +/* Bump up group capacity for forced packing of tasks */ + sgs.group_capacity += sds->group_capacity_bump; sds->total_load += sgs.group_load; sds->total_pwr += group->__cpu_power;
@@ -3768,6 +3788,12 @@ find_busiest_group(struct sched_domain *sd, int this_cpu,if (100 * sds.max_load <= sd->imbalance_pct * sds.this_load) goto out_balanced; +/* Push the upper limits for overload */ +if (sds.max_load <= (sds.busiest->__cpu_power + + sds.group_capacity_bump *SCHED_LOAD_SCALE ) / + sds.busiest->__cpu_power *SCHED_LOAD_SCALE ) + goto out_balanced; + sds.busiest_load_per_task /= sds.busiest_nr_running;if (sds.group_imb) sds.busiest_load_per_task =
Quoted 3 lines · Vaidyanathan Srinivasan
* Peter Zijlstra wanted more justifications for throttling at the core level. Throttling may be a resource management problem rather than scheduler/load balancer
No, I mandate that it be thermal management. Any other reason and you've got a NAK.
Re: [RFC PATCH v2 0/2] Saving power by cpu evacuationsched_max_capacity_pct=n
Quoted 8 lines · Peter Zijlstra
Quoted 3 lines · Vaidyanathan Srinivasan
* Peter Zijlstra wanted more justifications for throttling at the core level. Throttling may be a resource management problem rather than scheduler/load balancer
No, I mandate that it be thermal management. Any other reason and you've got a NAK.
Hi Peter,
Yes, I understand your objection. Your want throttling to be done for the purpose of thermal management only. The primary purpose for throttling should be thermal management (power savings may be a side-effect)
What I meant in the above comment was that the implementation for throttling could be solved using resource management framework, cpuset/cgroup rather than biasing the load balancer to avoid work on a particular core.
I am open to ideas for a clean and easy framework for core level throttling.
Thanks,
Vaidy
Re: [RFC PATCH v2 0/2] Saving power by cpu evacuationsched_max_capacity_pct=n
In reply to Peter ZijlstraQuoted 8 lines · Peter Zijlstra
Quoted 3 lines · Vaidyanathan Srinivasan
* Peter Zijlstra wanted more justifications for throttling at the core level. Throttling may be a resource management problem rather than scheduler/load balancer
No, I mandate that it be thermal management. Any other reason and you've got a NAK.
We've been discussing hard limits from the resource management view point. Bharata is working on a RFC that we should be able to share soon
Signature · 1 line
Balbir
Re: [RFC PATCH v2 0/2] Saving power by cpu evacuationsched_max_capacity_pct=n
Quoted 8 lines · Balbir Singh
Quoted 6 lines · Peter Zijlstra
Quoted 3 lines · Vaidyanathan Srinivasan
* Peter Zijlstra wanted more justifications for throttling at the core level. Throttling may be a resource management problem rather than scheduler/load balancer
Quoted 6 lines
Quoted 2 lines
No, I mandate that it be thermal management. Any other reason and you've got a NAK.
We've been discussing hard limits from the resource management view point. Bharata is working on a RFC that we should be able to share soon
Right, and you well know I dislike them.
Re: [RFC PATCH v2 0/2] Saving power by cpuevacuationsched_max_capacity_pct=n
Quoted 9 lines · Peter Zijlstra
Quoted 8 lines · Balbir Singh
Quoted 6 lines · Peter Zijlstra
Quoted 3 lines · Vaidyanathan Srinivasan
* Peter Zijlstra wanted more justifications for throttling at the core level. Throttling may be a resource management problem rather than scheduler/load balancer
Quoted 9 lines
Quoted 6 lines
Quoted 2 lines
No, I mandate that it be thermal management. Any other reason and you've got a NAK.
We've been discussing hard limits from the resource management view point. Bharata is working on a RFC that we should be able to share soon
Right, and you well know I dislike them.
Yes, but we hope to convince you with use cases and examples.
Signature · 1 line
Balbir
Quoted 3 lines · Vaidyanathan Srinivasan
* Using sched_mc=3,4,5 to evacuate 1,2,4 cores is completely non-intuitive and broken interface. Ingo wanted to see if we can model a global percentile tunable that would map to core throttling.
I have one request. CPU throttling is already a very well established term in the x86 world, refering to thermal throttling when the CPU overheats. This is implemented by ACPI and the CPU. It's always a very bad thing that should be avoided at all costs.
You seem to use it for something else. Can you please use a different term for that? Reusing the same word for something else is confusing.
Thanks,
-Andi
Signature · 1 line
[email protected] -- Speaking for myself only.
Quoted 9 lines · Andi Kleen
Quoted 3 lines · Vaidyanathan Srinivasan
* Using sched_mc=3,4,5 to evacuate 1,2,4 cores is completely non-intuitive and broken interface. Ingo wanted to see if we can model a global percentile tunable that would map to core throttling.
I have one request. CPU throttling is already a very well established term in the x86 world, refering to thermal throttling when the CPU overheats. This is implemented by ACPI and the CPU. It's always a very bad thing that should be avoided at all costs.
Its about avoiding that.
Quoted 6 lines · Peter Zijlstra
Quoted 5 lines · Andi Kleen
Quoted 3 lines · Vaidyanathan Srinivasan
* Using sched_mc=3,4,5 to evacuate 1,2,4 cores is completely non-intuitive and broken interface. Ingo wanted to see if we can model a global percentile tunable that would map to core throttling.
Quoted 6 lines
Quoted 4 lines
I have one request. CPU throttling is already a very well established term in the x86 world, refering to thermal throttling when the CPU overheats. This is implemented by ACPI and the CPU. It's always a very bad thing that should be avoided at all costs.
Its about avoiding that.
Hmm? Can you explain please? CPU throttling should only happen when your cooling system is broken in some way.
It's not a power saving feature, just a "don't make CPU melt" feature.
-Andi
Signature · 1 line
[email protected] -- Speaking for myself only.
Quoted 12 lines · Andi Kleen
Quoted 11 lines · Peter Zijlstra
Quoted 9 lines · Andi Kleen
Quoted 3 lines · Vaidyanathan Srinivasan
* Using sched_mc=3,4,5 to evacuate 1,2,4 cores is completely non-intuitive and broken interface. Ingo wanted to see if we can model a global percentile tunable that would map to core throttling.
I have one request. CPU throttling is already a very well established term in the x86 world, refering to thermal throttling when the CPU overheats. This is implemented by ACPI and the CPU. It's always a very bad thing that should be avoided at all costs.
Quoted 8 lines
Quoted 1 line
Its about avoiding that.
Hmm? Can you explain please? CPU throttling should only happen when your cooling system is broken in some way.
It's not a power saving feature, just a "don't make CPU melt" feature.
From what I've been told its popular to over-commit the cooling capacity
in a rack, so that a number of servers can run at full thermal capacity but not all.
I've also been told that hardware sucks at throttling, therefore people want to fix the OS so as to limit the thermal capacity and avoid the hardware throttle from kicking in, whilst still not exceeding the rack capacity or similar nonsense.
Quoted 3 lines
Quoted 1 line
From what I've been told its popular to over-commit the cooling capacity
in a rack, so that a number of servers can run at full thermal capacity but not all.
Yes. But in this case you don't want to use throttling, you want to use p-states which actually safe power unlike throttling.
Quoted 1 line
I've also been told that hardware sucks at throttling,
Throttling is not really something you should use in normal operation, it's just a emergency measure. For that it works quite well, but you really don't want it in normal operation.
Quoted 4 lines
therefore people
want to fix the OS so as to limit the thermal capacity and avoid the hardware throttle from kicking in, whilst still not exceeding the rack capacity or similar nonsense.
Yes that's fine and common, but you actually need to save power for this, which throttling doesn't do.
My understanding this work is a extension of the existing sched_mc_power_savings features that tries to be optionally more aggressive to keep complete package idle so that package level power saving kicks in.
I'm just requesting that they don't call that throttling.
-Andi
Signature · 1 line
[email protected] -- Speaking for myself only.
Quoted 21 lines · Andi Kleen
Quoted 3 lines
Quoted 1 line
From what I've been told its popular to over-commit the cooling capacity
in a rack, so that a number of servers can run at full thermal capacity but not all.
Yes. But in this case you don't want to use throttling, you want to use p-states which actually safe power unlike throttling.
Quoted 1 line
I've also been told that hardware sucks at throttling,
Throttling is not really something you should use in normal operation, it's just a emergency measure. For that it works quite well, but you really don't want it in normal operation.
Quoted 4 lines
therefore people
want to fix the OS so as to limit the thermal capacity and avoid the hardware throttle from kicking in, whilst still not exceeding the rack capacity or similar nonsense.Yes that's fine and common, but you actually need to save power for this, which throttling doesn't do.
Quoted 6 lines
My understanding this work is a extension of the existing
sched_mc_power_savingsfeatures that tries to be optionally more aggressive to keep complete package idle so that package level power saving kicks in.I'm just requesting that they don't call that throttling.
Ah no, this work differs in that regard in that it actually 'generates' idle time, instead of optimizing idle time.
Therefore it takes actual cpu time away from real work, which is throttling. Granted, one could call it limiting or similar, but throttling is a correct name.
Quoted 8 lines
Quoted 8 lines
Yes that's fine and common, but you actually need to save power for this, which throttling doesn't do.
My understanding this work is a extension of the existing
sched_mc_power_savingsfeatures that tries to be optionally more aggressive to keep complete package idle so that package level power saving kicks in.
Quoted 4 lines
Quoted 1 line
I'm just requesting that they don't call that throttling.
Ah no, this work differs in that regard in that it actually 'generates' idle time, instead of optimizing idle time.
That is what i meant with "more aggressive to keep complete packages idle" above.
Quoted 4 lines
Therefore it takes actual cpu time away from real work, which is throttling. Granted, one could call it limiting or similar, but throttling is a correct name.
That will be always ongoing confusion with the existing established term.
If you really need to call it throttling use "scheduler throttling" or something like that, but a different word would be better.
-Andi
Signature · 1 line
[email protected] -- Speaking for myself only.
Quoted 8 lines · Andi Kleen
Quoted 8 lines
Quoted 8 lines
Yes that's fine and common, but you actually need to save power for this, which throttling doesn't do.
My understanding this work is a extension of the existing
sched_mc_power_savingsfeatures that tries to be optionally more aggressive to keep complete package idle so that package level power saving kicks in.
Quoted 7 lines
Quoted 4 lines
Quoted 1 line
I'm just requesting that they don't call that throttling.
Ah no, this work differs in that regard in that it actually 'generates' idle time, instead of optimizing idle time.
That is what i meant with "more aggressive to keep complete packages idle" above.
Hi Andi,
There is a difference in the framework as Peter has mentioned, we are trying to create idle times by forcefully reducing work. From an end-user point of view, this can be seen as a logical extension of sched_mc_power_savings... v1 of the RFC extends the framework.
However Ingo suggested that the knob is not intuitive and hence I have tried to switch to a percentage knob sched_max_capacity_pct.
I am interested in an easy, simple and intuitive framework to evacuate cores which may imply forcefully reducing (throttling) work.
Quoted 9 lines
Quoted 3 lines
Therefore it takes actual cpu time away from real work, which is throttling. Granted, one could call it limiting or similar, but throttling is a correct name.
That will be always ongoing confusion with the existing established term.
If you really need to call it throttling use "scheduler throttling" or something like that, but a different word would be better.
I think 'scheduler throttling' is good so that we avoid the term 'CPU throttling' or core throttling. I had named this cpu evacuation or core evacuation just to avoid confusion with hardware throttling.
--Vaidy
Quoted 3 lines
I think 'scheduler throttling' is good so that we avoid the term 'CPU throttling' or core throttling. I had named this cpu evacuation or core evacuation just to avoid confusion with hardware throttling.
Evacuation sounds good, although shouldn't it be package or socket evacuation?
-Andi
Signature · 1 line
[email protected] -- Speaking for myself only.
Quoted 6 lines · Andi Kleen
Quoted 3 lines
I think 'scheduler throttling' is good so that we avoid the term 'CPU throttling' or core throttling. I had named this cpu evacuation or core evacuation just to avoid confusion with hardware throttling.
Evacuation sounds good, although shouldn't it be package or socket evacuation?
Lets start with 'core evacuation' since that seems to be the lowest granularity on a threaded system. We certainly want the framework to provide socket/package and node evacuation on much larger systems.
--Vaidy
Quoted 6 lines · Andi Kleen
Quoted 3 lines
Quoted 1 line
From what I've been told its popular to over-commit the cooling capacity
in a rack, so that a number of servers can run at full thermal capacity but not all.
Yes. But in this case you don't want to use throttling, you want to use p-states which actually safe power unlike throttling.
One of the design points for the discussion is to bring in C-States into the equation. As you have mentioned today we can effectively use P-States to reduce core frequency and thereby reduce average power and heat. With the introduction of very low power deep sleep states in the processor, C-States can provide substantial power savings apart from just P-State based methods. Forcefully idling cores will lead to exploitation of C-States and their power savings benefits.
As mentioned earlier, cpu throttling as it exist today should not be used in normal operating conditions. However exploiting P-States and C-States as two control variables, the system can be made to operate at various power (thermal) and performance points.
Quoted 7 lines
Quoted 2 lines
I've also been told that hardware sucks at throttling,
Throttling is not really something you should use in normal operation, it's just a emergency measure. For that it works quite well, but you really don't want it in normal operation.
Quoted 7 lines
Quoted 4 lines
therefore people
want to fix the OS so as to limit the thermal capacity and avoid the hardware throttle from kicking in, whilst still not exceeding the rack capacity or similar nonsense.Yes that's fine and common, but you actually need to save power for this, which throttling doesn't do.
Reducing work, scheduling them smartly in the OS can greatly save power as compared to throttling in hardware in order to reduce power or heat.
Quoted 4 lines
My understanding this work is a extension of the existing
sched_mc_power_savingsfeatures that tries to be optionally more aggressive to keep complete package idle so that package level power saving kicks in.
Scheduling work smartly (power efficiently) is part of the sched_mc_power_savings framework, while this RFC/discussion is around reducing work or forcing idle times but at a granularity of cores/packages to provide maximum power/thermal benefits.
--Vaidy
Quoted 13 lines · Andi Kleen
Quoted 3 lines
Quoted 1 line
From what I've been told its popular to over-commit the cooling capacity
in a rack, so that a number of servers can run at full thermal capacity but not all.
Yes. But in this case you don't want to use throttling, you want to use p-states which actually safe power unlike throttling.
Quoted 1 line
I've also been told that hardware sucks at throttling,
Throttling is not really something you should use in normal operation, it's just a emergency measure. For that it works quite well, but you really don't want it in normal operation.
Quoted 7 lines
Quoted 4 lines
therefore people
want to fix the OS so as to limit the thermal capacity and avoid the hardware throttle from kicking in, whilst still not exceeding the rack capacity or similar nonsense.Yes that's fine and common, but you actually need to save power for this, which throttling doesn't do.
Actually throttling will lower power consumption at any given moment (not power consumption for any given task!) and will keep your rack from melting.
But I don't see why it is neccessary to evacuate cores for this. Why not just schedule special task that enters C3 instead of computing?
That was what I planned to do on athlon 900 (1 core) with broken fan...
For what you are doing, cpu hotplug seems more suitable. Can you
enhance it so that it is fast enough for you?
Pavel
Signature · 2 lines
(english) http://www.livejournal.com/~pavelmachek (cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
Quoted 19 lines · Pavel Machek
Quoted 18 lines · Andi Kleen
Quoted 3 lines
Quoted 1 line
From what I've been told its popular to over-commit the cooling capacity
in a rack, so that a number of servers can run at full thermal capacity but not all.
Yes. But in this case you don't want to use throttling, you want to use p-states which actually safe power unlike throttling.
Quoted 1 line
I've also been told that hardware sucks at throttling,
Throttling is not really something you should use in normal operation, it's just a emergency measure. For that it works quite well, but you really don't want it in normal operation.
Quoted 4 lines
therefore people
want to fix the OS so as to limit the thermal capacity and avoid the hardware throttle from kicking in, whilst still not exceeding the rack capacity or similar nonsense.
Quoted 6 lines
Quoted 2 lines
Yes that's fine and common, but you actually need to save power for this, which throttling doesn't do.
Actually throttling will lower power consumption at any given moment (not power consumption for any given task!) and will keep your rack from melting.
Yes, we want to reduce overall power consumption.
Quoted 2 lines
But I don't see why it is neccessary to evacuate cores for this. Why not just schedule special task that enters C3 instead of computing?
This is what essentially happens in the load balancer approach. Not scheduling on a particular core will run the scheduler's idle task that will transition the core to lowest power state. Pinning a user space task and using special driver to hold the core in C3 state will break scheduling fairness. At this point the application decides when to give the core back to scheduler.
Quoted 5 lines
That was what I planned to do on athlon 900 (1 core) with broken fan...
For what you are doing, cpu hotplug seems more suitable. Can you
enhance it so that it is fast enough for you?
Yes cpu hotplug framework can be used. That is definitely an alternative to this approach. However in the case of cpuhotplug, the evacuation is directed to a particular core which may affect user space affinity and cpusets. But in this case we can limit the overall system capacity, like run at most 7 cores at a time in an 8 core system, but we actually don't need to care which particular core is 'forced to idle' at any given point in time.
Further discussion regarding this can be found in the following
thread: http://lkml.org/lkml/2009/5/19/54
--Vaidy
Hi!
Quoted 9 lines
Quoted 2 lines
But I don't see why it is neccessary to evacuate cores for this. Why not just schedule special task that enters C3 instead of computing?
This is what essentially happens in the load balancer approach. Not scheduling on a particular core will run the scheduler's idle task that will transition the core to lowest power state. Pinning a user space task and using special driver to hold the core in C3 state will break scheduling fairness. At this point the application decides when to give the core back to scheduler.
Why would it break scheduling fairness? You just schedule "realtime" task that does C3 instead of computation. The behaviour is very similar to "normal" realtime task. Why would it break scheduler?
Pavel
Signature · 2 lines
(english) http://www.livejournal.com/~pavelmachek (cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html