sched: a rough proposal to enable power saving in scheduler

Since there is no power saving consideration in scheduler CFS, I has a very rough idea for enabling a new power saving schema in CFS.

It bases on the following assumption:
1, If there are many task crowd in system, just let few domain cpus running and let other cpus idle can not save power. Let all cpu take the load, finish tasks early, and then get into idle. will save more power and have better user experience.

2, schedule domain, schedule group perfect match the hardware, and the power consumption unit. So, pull tasks out of a domain means potentially this power consumption unit idle.

So, according Peter mentioned in commit 8e7fbcbc22c(sched: Remove stale power aware scheduling), this proposal will adopt the sched_balance_policy concept and use 2 kind of policy: performance, power.

And in scheduling, 2 place will care the policy, load_balance() and in task fork/exec: select_task_rq_fair().

Here is some pseudo code try to explain the proposal behaviour in load_balance() and select_task_rq_fair();

load_balance() {
	update_sd_lb_stats(); //get busiest group, idlest group data.

	if (sd->nr_running > sd's capacity) {
		//power saving policy is not suitable for
		//this scenario, it runs like performance policy
		mv tasks from busiest cpu in busiest group to
		idlest 	cpu in idlest group;
	} else {// the sd has enough capacity to hold all tasks.
		if (sg->nr_running > sg's capacity) {
			//imbalanced between groups
			if (schedule policy == performance) {
				//when 2 busiest group at same busy
				//degree, need to prefer the one has
				// softest group??
				move tasks from busiest group to
					idletest group;
			} else if (schedule policy == power)
				move tasks from busiest group to
				idlest group until busiest is just full
				of capacity.
				//the busiest group can balance
				//internally after next time LB,
		} else {
			//all groups has enough capacity for its tasks.
			if (schedule policy == performance)
				//all tasks may has enough cpu
				//resources to run,
				//mv tasks from busiest to idlest group?
				//no, at this time, it's better to keep
				//the task on current cpu.
				//so, it is maybe better to do balance
				//in each of groups
				for_each_imbalance_groups()
					move tasks from busiest cpu to
					idlest cpu in each of groups;
			else if (schedule policy == power) {
				if (no hard pin in idlest group)
					mv tasks from idlest group to
					busiest until busiest full.
				else
					mv unpin tasks to the biggest
					hard pin group.
			}
		}
	}
}

select_task_rq_fair()

{
	for_each_domain(cpu, tmp) {
		if (policy == power && tmp_has_capacity &&
			 tmp->flags & sd_flag) {
			sd = tmp;
			//It is fine to got cpu in the domain
			break;
		}
	}

	while(sd) {
		if policy == power
			find_busiest_and_capable_group()
		else
			find_idlest_group();
		if (!group) {
			sd = sd->child;
			continue;
		}
		...
	}
}

sub proposal:
1, If it's possible to balance task on idlest cpu not appointed 'balance cpu'. If so, it may can reduce one more time balancing.
The idlest cpu can prefer the new idle cpu; and is the least load cpu; 2, se or task load is good for running time setting.
but it should the second basis in load balancing. The first basis of LB is running tasks' number in group/cpu. Since whatever of the weight of groups is, if the tasks number is less than cpu number, the group is still has capacity to take more tasks. (will consider the SMT cpu power or other big/little cpu capacity on ARM.)

unsolved issues:
1, like current scheduler, it didn't handled cpu affinity well in load_balance.
2, task group that isn't consider well in this rough proposal.

It isn't consider well and may has mistaken . So just share my ideas and hope it become better and workable in your comments and discussion.

Thanks
Alex

Any comments for this rough proposal, specially for the assumptions?

Since there is no power saving consideration in scheduler CFS, I has a very rough idea for enabling a new power saving schema in CFS.

It bases on the following assumption:
1, If there are many task crowd in system, just let few domain cpus running and let other cpus idle can not save power. Let all cpu take the load, finish tasks early, and then get into idle. will save more power and have better user experience.

2, schedule domain, schedule group perfect match the hardware, and the power consumption unit. So, pull tasks out of a domain means potentially this power consumption unit idle.

So, according Peter mentioned in commit 8e7fbcbc22c(sched: Remove stale power aware scheduling), this proposal will adopt the sched_balance_policy concept and use 2 kind of policy: performance, power.

And in scheduling, 2 place will care the policy, load_balance() and in task fork/exec: select_task_rq_fair().

Here is some pseudo code try to explain the proposal behaviour in load_balance() and select_task_rq_fair();

load_balance() {
	update_sd_lb_stats(); //get busiest group, idlest group data.

	if (sd->nr_running > sd's capacity) {
		//power saving policy is not suitable for
		//this scenario, it runs like performance policy
		mv tasks from busiest cpu in busiest group to
		idlest 	cpu in idlest group;

	} else {// the sd has enough capacity to hold all tasks.
		if (sg->nr_running > sg's capacity) {
			//imbalanced between groups
			if (schedule policy == performance) {
				//when 2 busiest group at same busy
				//degree, need to prefer the one has
				// softest group??
				move tasks from busiest group to
					idletest group;

			} else if (schedule policy == power)
				move tasks from busiest group to
				idlest group until busiest is just full
				of capacity.
				//the busiest group can balance
				//internally after next time LB,

		} else {
			//all groups has enough capacity for its tasks.
			if (schedule policy == performance)
				//all tasks may has enough cpu
				//resources to run,
				//mv tasks from busiest to idlest group?
				//no, at this time, it's better to keep
				//the task on current cpu.
				//so, it is maybe better to do balance
				//in each of groups
				for_each_imbalance_groups()
					move tasks from busiest cpu to
					idlest cpu in each of groups;
			else if (schedule policy == power) {
				if (no hard pin in idlest group)
					mv tasks from idlest group to

					busiest until busiest full.
				else
					mv unpin tasks to the biggest
					hard pin group.
			}
		}
	}
}

sub proposal:
1, If it's possible to balance task on idlest cpu not appointed 'balance cpu'. If so, it may can reduce one more time balancing.
The idlest cpu can prefer the new idle cpu; and is the least load cpu; 2, se or task load is good for running time setting.
but it should the second basis in load balancing. The first basis of LB is running tasks' number in group/cpu. Since whatever of the weight of groups is, if the tasks number is less than cpu number, the group is still has capacity to take more tasks. (will consider the SMT cpu power or other big/little cpu capacity on ARM.)

unsolved issues:
1, like current scheduler, it didn't handled cpu affinity well in load_balance.

2, task group that isn't consider well in this rough proposal.

It isn't consider well and may has mistaken . So just share my ideas and hope it become better and workable in your comments and discussion.

Quoted 2 lines · Alex Shi

Since there is no power saving consideration in scheduler CFS, I has a very rough idea for enabling a new power saving schema in CFS.

Adding Thomas, he always delights poking holes in power schemes.

Quoted 5 lines

It bases on the following assumption:
1, If there are many task crowd in system, just let few domain cpus running and let other cpus idle can not save power. Let all cpu take the load, finish tasks early, and then get into idle. will save more power and have better user experience.

I'm not sure this is a valid assumption. I've had it explained to me by various people that race-to-idle isn't always the best thing. It has to do with the cost of switching power states and the duration of execution and other such things.

So I'd leave the currently implemented scheme as performance, and I don't think the above describes the current state.

Quoted 6 lines

			} else if (schedule policy == power)
				move tasks from busiest group to
				idlest group until busiest is just full
				of capacity.
				//the busiest group can balance
				//internally after next time LB,

There's another thing we need to do, and that is collect tasks in a minimal amount of power domains.

Yay, ideally we'd also provide a 3rd option: auto, which simply switches between the two based on AC/BAT, UPS

I'm not sure this is a valid assumption. I've had it explained to me by various people that race-to-idle isn't always the best thing.

Since there is no power saving consideration in scheduler CFS, I has a very rough idea for enabling a new power saving schema in CFS.

It bases on the following assumption:
1, If there are many task crowd in system, just let few domain cpus running and let other cpus idle can not save power. Let all cpu take the load, finish tasks early, and then get into idle. will save more power and have better user experience.

2, schedule domain, schedule group perfect match the hardware, and the power consumption unit. So, pull tasks out of a domain means potentially this power consumption unit idle.

select_task_rq_fair()

{
	for_each_domain(cpu, tmp) {
		if (policy == power && tmp_has_capacity &&
			 tmp->flags & sd_flag) {
			sd = tmp;
			//It is fine to got cpu in the domain
			break;
		}
	}

	while(sd) {
		if policy == power
			find_busiest_and_capable_group()

Quoted 2 lines · Peter Zijlstra

Yay, ideally we'd also provide a 3rd option: auto, which simply switches between the two based on AC/BAT, UPS

nooooooo!

anything but this.
if anyone thinks that AC/Battery matters for power sensitivity... they need to go talk to a datacenter operator ;-)

seriously, there are possibly many ways to have a power/performance preference..,. but AC/battery is a very very poor one.

Quoted 12 lines · Peter Zijlstra

Quoted 2 lines · Alex Shi

Since there is no power saving consideration in scheduler CFS, I has a very rough idea for enabling a new power saving schema in CFS.

Adding Thomas, he always delights poking holes in power schemes.

Quoted 5 lines

It bases on the following assumption:
1, If there are many task crowd in system, just let few domain cpus running and let other cpus idle can not save power. Let all cpu take the load, finish tasks early, and then get into idle. will save more power and have better user experience.

Quoted 4 lines

I'm not sure this is a valid assumption. I've had it explained to me by various people that race-to-idle isn't always the best thing. It has to do with the cost of switching power states and the duration of execution and other such things.

I think what he means here is that we might want to let all cores on the node (i.e., domain) finish and then power down the whole node which should bring much more power savings than letting a subset of the cores idle. Alex?

Quoted 10 lines

So I'd leave the currently implemented scheme as performance, and I don't think the above describes the current state.

Quoted 6 lines

			} else if (schedule policy == power)
				move tasks from busiest group to
				idlest group until busiest is just full
				of capacity.
				//the busiest group can balance
				//internally after next time LB,

Quoted 2 lines

There's another thing we need to do, and that is collect tasks in a minimal amount of power domains.

Yep.

Btw, what heuristic would tell here when a domain overflows and another needs to get woken? Combined load of the whole domain?

And if I absolutely positively don't want a node to wake up, do I hotplug its cores off or are we going to have a way to tell the scheduler to overcommit the non-idle domains and spread the tasks only among them.

I'm thinking of short bursts here where it would be probably beneficial to let the tasks rather wait runnable for a while then wake up the next node and waste power...

Quoted 4 lines · Arjan van de Ven

Quoted 2 lines · Peter Zijlstra

Yay, ideally we'd also provide a 3rd option: auto, which simply switches between the two based on AC/BAT, UPS

Quoted 5 lines

nooooooo!

anything but this.
if anyone thinks that AC/Battery matters for power sensitivity... they need to go talk to a datacenter operator ;-)

Servers in a datacenter have battery?

Quoted 4 lines

seriously, there are possibly many ways to have a power/performance preference..,. but AC/battery is a very very poor one.

Do expand..

Ideally the scheduler builds up some history of the typical run duration of the task (with a bias to more recent runs). But... even then, the past is only a poor predictor for the future.

Quoted 3 lines · Arjan van de Ven

Ideally the scheduler builds up some history of the typical run duration of the task (with a bias to more recent runs). But... even then, the past is only a poor predictor for the future.

PJTs patches do this. But yes, a crystal-ball instruction which completes in a single cycle would be most appreciated :-)

How do you plan to test this power saving scheme? Using powertop? Or, is there any other tools?

Quoted 1 line

Servers in a datacenter have battery?

they have AC, and sometimes a battery called "UPS". DC is getting much more prevalent in datacenters in general.

Quoted 5 lines

Quoted 3 lines

seriously, there are possibly many ways to have a power/performance preference..,. but AC/battery is a very very poor one.

Do expand..

The easy cop-out is provide the sysadmin a slider.
The slightly less easy one is to (and we're taking this approach in the new P state code we're working on) say "in the default setting, we're going to sacrifice up to 5% performance from peak to give you the best power savings within that performance loss budget" (with a slider that can give you 0%, 2 1/2% 5% and 10%)

on Intel PCs and servers, there usually is a bios switch/setting for this (there is a setting the bios does to the CPU, and we can read that. Not all bioses expose this setting to the end user). We could take clue from what was set there.

power aware scheduling), this proposal will adopt the sched_balance_policy concept and use 2 kind of policy: performance, power.

Yay, ideally we'd also provide a 3rd option: auto, which simply switches between the two based on AC/BAT, UPS status and simple things like that.

Quoted 5 lines · Alex Shi

It bases on the following assumption:
1, If there are many task crowd in system, just let few domain cpus running and let other cpus idle can not save power. Let all cpu take the load, finish tasks early, and then get into idle. will save more power and have better user experience.

I'm not sure this is a valid assumption. I've had it explained to me by various people that race-to-idle isn't always the best thing. It has to do with the cost of switching power states and the duration of execution and other such things.

This all sounds far too complicated.. we're talking about simple spreading and packing balancers without deep arch knowledge and knobs, we couldn't possibly evaluate anything like that.

I was really more thinking of something useful for the laptops out there, when they pull the power cord it makes sense to try and keep CPUs asleep until the one that's awake is saturated.

Quoted 7 lines · Peter Zijlstra

Quoted 5 lines · Alex Shi

It bases on the following assumption:
1, If there are many task crowd in system, just let few domain cpus running and let other cpus idle can not save power. Let all cpu take the load, finish tasks early, and then get into idle. will save more power and have better user experience.

Quoted 4 lines

I'm not sure this is a valid assumption. I've had it explained to me by various people that race-to-idle isn't always the best thing. It has to do with the cost of switching power states and the duration of execution and other such things.

This is affected by Intel's implementation - if there's a single active

Quoted 2 lines · Rakib Mullick

How do you plan to test this power saving scheme? Using powertop? Or, is there any other tools?

We should start out simple enough that we can validate it by looking at task placement by hand, eg. 4 tasks on a dual socket quad-core, should only keep one socket awake.

We can also add an power aware evaluator to Linsched (another one of those things that needs getting sorted).

And yeah, someone running with a power meter is of course king.

_BUT_ we shouldn't go off the wall with power meters as that very quickly gets very specific to the system being measured.

We should really keep this thing as simple as possible while still providing some benefit for all various architectures without tons of per arch knobs and knowhow.

The easy cop-out is provide the sysadmin a slider.
The slightly less easy one is to (and we're taking this approach in the new P state code we're working on) say "in the default setting, we're going to sacrifice up to 5% performance from peak to give you the best power savings within that performance loss budget" (with a slider that can give you 0%, 2 1/2% 5% and 10%)

Quoted 6 lines · Arjan van de Ven

The easy cop-out is provide the sysadmin a slider.
The slightly less easy one is to (and we're taking this approach in the new P state code we're working on) say "in the default setting, we're going to sacrifice up to 5% performance from peak to give you the best power savings within that performance loss budget" (with a slider that can give you 0%, 2 1/2% 5% and 10%)

On a related note, I am looking at the c-state menu governor.

We seem to have issues there, with Linux often going into a much deeper C state than warranted, which can lead to a fairly steep performance penalty for some workloads.

The idea Matthew and I have is simply planning for a shorter sleep period (discarding the outliers to the high end in the function once known as detect_repeating_patterns), and going to a deeper C state if we have significantly overslept.

The new estimation code is easy, but for the past days I have been looking through the timer code to figure out how such a timer could fire, and how we could recognize it without it looking like a normal wakeup, if we do not end up accidentally waking up another CPU, etc...

Quoted 2 lines · Alex Shi

Since there is no power saving consideration in scheduler CFS, I has a very rough idea for enabling a new power saving schema in CFS.

Adding Thomas, he always delights poking holes in power schemes.

Quoted 5 lines

It bases on the following assumption:
1, If there are many task crowd in system, just let few domain cpus running and let other cpus idle can not save power. Let all cpu take the load, finish tasks early, and then get into idle. will save more power and have better user experience.

I'm not sure this is a valid assumption. I've had it explained to me by various people that race-to-idle isn't always the best thing. It has to do with the cost of switching power states and the duration of execution and other such things.

Quoted 4 lines

2, schedule domain, schedule group perfect match the hardware, and the power consumption unit. So, pull tasks out of a domain means potentially this power consumption unit idle.

I'm not sure I understand what you're saying, sorry.

Quoted 4 lines

So, according Peter mentioned in commit 8e7fbcbc22c(sched: Remove stale power aware scheduling), this proposal will adopt the sched_balance_policy concept and use 2 kind of policy: performance, power.

Yay, ideally we'd also provide a 3rd option: auto, which simply switches between the two based on AC/BAT, UPS status and simple things like that. But this seems like a later concern, you have to have something to pick between before you can pick :-)

Quoted 3 lines

And in scheduling, 2 place will care the policy, load_balance() and in task fork/exec: select_task_rq_fair().

ack

Quoted 3 lines

Here is some pseudo code try to explain the proposal behaviour in load_balance() and select_task_rq_fair();

Oh man.. A few words outlining the general idea would've been nice.

Quoted 8 lines

load_balance() {
	update_sd_lb_stats(); //get busiest group, idlest group data.

	if (sd->nr_running > sd's capacity) {
		//power saving policy is not suitable for
		//this scenario, it runs like performance policy
		mv tasks from busiest cpu in busiest group to
		idlest 	cpu in idlest group;

Once upon a time we talked about adding a factor to the capacity for this. So say you'd allow 2*capacity before overflowing and waking another power group.

But I think we should not go on nr_running here, PJTs per-entity load tracking stuff gives us much better measures -- also, repost that series already Paul! :-)

Also, I'm not sure this is entirely correct, the thing you want to do for power aware stuff is to minimize the number of active power domains, this means you don't want idlest, you want least busy non-idle.

Quoted 10 lines

	} else {// the sd has enough capacity to hold all tasks.
		if (sg->nr_running > sg's capacity) {
			//imbalanced between groups
			if (schedule policy == performance) {
				//when 2 busiest group at same busy
				//degree, need to prefer the one has
				// softest group??
				move tasks from busiest group to
					idletest group;

So I'd leave the currently implemented scheme as performance, and I don't think the above describes the current state.

Quoted 7 lines

			} else if (schedule policy == power)
				move tasks from busiest group to
				idlest group until busiest is just full
				of capacity.
				//the busiest group can balance
				//internally after next time LB,

There's another thing we need to do, and that is collect tasks in a minimal amount of power domains. The old code (that got deleted) did something like that, you can revive some of the that code if needed -- I just killed everything to be able to start with a clean slate.

Quoted 18 lines

		} else {
			//all groups has enough capacity for its tasks.
			if (schedule policy == performance)
				//all tasks may has enough cpu
				//resources to run,
				//mv tasks from busiest to idlest group?
				//no, at this time, it's better to keep
				//the task on current cpu.
				//so, it is maybe better to do balance
				//in each of groups
				for_each_imbalance_groups()
					move tasks from busiest cpu to
					idlest cpu in each of groups;
			else if (schedule policy == power) {
				if (no hard pin in idlest group)
					mv tasks from idlest group to

Quoted 8 lines

					busiest until busiest full.
				else
					mv unpin tasks to the biggest
					hard pin group.
			}
		}
	}
}

OK, so you only start to group later.. I think we can do better than that.

Quoted 12 lines

sub proposal:
1, If it's possible to balance task on idlest cpu not appointed 'balance cpu'. If so, it may can reduce one more time balancing.
The idlest cpu can prefer the new idle cpu; and is the least load cpu; 2, se or task load is good for running time setting.
but it should the second basis in load balancing. The first basis of LB is running tasks' number in group/cpu. Since whatever of the weight of groups is, if the tasks number is less than cpu number, the group is still has capacity to take more tasks. (will consider the SMT cpu power or other big/little cpu capacity on ARM.)

Ah, no we shouldn't balance on nr_running, but on the amount of time consumed. Imagine two tasks being woken at the same time, both tasks will only run a fraction of the available time, you don't want this to exceed your capacity because ran back to back the one cpu will still be mostly idle.

What you want it to keep track of a per-cpu utilization level (inverse of idle-time) and using PJTs per-task runnable avg see if placing the new task on will exceed the utilization limit.

I think some of the Linaro people actually played around with this, Vincent?

Quoted 3 lines

unsolved issues:
1, like current scheduler, it didn't handled cpu affinity well in load_balance.

cpu affinity is always 'fun'.. while there's still a few fun sites in the current load-balancer we do better than we did a while ago.

Quoted 1 line

2, task group that isn't consider well in this rough proposal.

You mean the cgroup mess?

Quoted 3 lines

It isn't consider well and may has mistaken . So just share my ideas and hope it become better and workable in your comments and discussion.

Very simplistically the current scheme is a 'spread' the load scheme (SD_PREFER_SIBLING if you will). We spread load to maximize per-task cache and cpu power.

The power scheme should be a 'pack' scheme, where we minimize the active power domains.

One way to implement this is to keep track of an active and under-utilized power domain (the target) and fail the regular (pull) load-balance for all cpus not in that domain. For the cpu that are in that domain we'll have find_busiest select from all other under-utilized domains pulling tasks to fill our target, once full, we pick a new target, goto 1.

Quoted 12 lines · Peter Zijlstra

Quoted 2 lines · Alex Shi

Since there is no power saving consideration in scheduler CFS, I has a very rough idea for enabling a new power saving schema in CFS.

Adding Thomas, he always delights poking holes in power schemes.

Quoted 5 lines

It bases on the following assumption:
1, If there are many task crowd in system, just let few domain cpus running and let other cpus idle can not save power. Let all cpu take the load, finish tasks early, and then get into idle. will save more power and have better user experience.

Quoted 4 lines

I'm not sure this is a valid assumption. I've had it explained to me by various people that race-to-idle isn't always the best thing. It has to do with the cost of switching power states and the duration of execution and other such things.

I think what he means here is that we might want to let all cores on the node (i.e., domain) finish and then power down the whole node which should bring much more power savings than letting a subset of the cores idle. Alex?

[ … ]

Quoted 10 lines

So I'd leave the currently implemented scheme as performance, and I don't think the above describes the current state.

Quoted 6 lines

			} else if (schedule policy == power)
				move tasks from busiest group to
				idlest group until busiest is just full
				of capacity.
				//the busiest group can balance
				//internally after next time LB,

Quoted 2 lines

There's another thing we need to do, and that is collect tasks in a minimal amount of power domains.

Yep.

Btw, what heuristic would tell here when a domain overflows and another needs to get woken? Combined load of the whole domain?

And if I absolutely positively don't want a node to wake up, do I hotplug its cores off or are we going to have a way to tell the scheduler to overcommit the non-idle domains and spread the tasks only among them.

I'm thinking of short bursts here where it would be probably beneficial to let the tasks rather wait runnable for a while then wake up the next node and waste power...

Thanks.

Quoted 18 lines · Borislav Petkov

Quoted 12 lines · Peter Zijlstra

Quoted 2 lines · Alex Shi

Since there is no power saving consideration in scheduler CFS, I has a very rough idea for enabling a new power saving schema in CFS.

Adding Thomas, he always delights poking holes in power schemes.

Quoted 5 lines

It bases on the following assumption:
1, If there are many task crowd in system, just let few domain cpus running and let other cpus idle can not save power. Let all cpu take the load, finish tasks early, and then get into idle. will save more power and have better user experience.

Quoted 4 lines

I'm not sure this is a valid assumption. I've had it explained to me by various people that race-to-idle isn't always the best thing. It has to do with the cost of switching power states and the duration of execution and other such things.

Quoted 4 lines

I think what he means here is that we might want to let all cores on the node (i.e., domain) finish and then power down the whole node which should bring much more power savings than letting a subset of the cores idle. Alex?

Sure we can do that.

Quoted 23 lines

Quoted 10 lines

So I'd leave the currently implemented scheme as performance, and I don't think the above describes the current state.

Quoted 6 lines

			} else if (schedule policy == power)
				move tasks from busiest group to
				idlest group until busiest is just full
				of capacity.
				//the busiest group can balance
				//internally after next time LB,

Quoted 2 lines

There's another thing we need to do, and that is collect tasks in a minimal amount of power domains.

Yep.

Btw, what heuristic would tell here when a domain overflows and another needs to get woken? Combined load of the whole domain?

And if I absolutely positively don't want a node to wake up, do I hotplug its cores off or are we going to have a way to tell the scheduler to overcommit the non-idle domains and spread the tasks only among them.

Quoted 3 lines

I'm thinking of short bursts here where it would be probably beneficial to let the tasks rather wait runnable for a while then wake up the next node and waste power...

I was thinking of a utilization measure made of per-task weighted runnable averages. This should indeed cover that case and we'll overflow when on average there is no (significant) idle time over a period longer than the averaging period.

Anyway, I'm not too set on this and I'm very sure we can tweak this ad infinitum, so starting with something relatively simple that works for most is preferred.

As already stated, I think some of the Linaro people actually played around with something like this based on PJTs patches.

Quoted 9 lines · Alex Shi

Since there is no power saving consideration in scheduler CFS, I has a very rough idea for enabling a new power saving schema in CFS.

It bases on the following assumption:
1, If there are many task crowd in system, just let few domain cpus running and let other cpus idle can not save power. Let all cpu take the load, finish tasks early, and then get into idle. will save more power and have better user experience.

This assumption indirectly point towards the scheme when performance is enabled, isn't it? Cause you're trying to spread the load equally amongst all the CPUs.

Quoted 4 lines

select_task_rq_fair()
{

Quoted 5 lines

	for_each_domain(cpu, tmp) {
		if (policy == power && tmp_has_capacity &&
			 tmp->flags & sd_flag) {
			sd = tmp;
			//It is fine to got cpu in the domain

Quoted 5 lines

			break;
		}
	}

	while(sd) {

Quoted 1 line

find_busiest_and_capable_group()

I'm not sure what find_busiest_and_capable_group() would really be, it seems it'll find the busiest and capable group, but isn't it a conflict with the first assumption you proposed on your proposal?

Thanks,
Rakib.

Quoted 2 lines · Alex Shi

power aware scheduling), this proposal will adopt the sched_balance_policy concept and use 2 kind of policy: performance, power.

Are there workloads in which "power" might provide more performance than "performance"? If so, don't use these terms.

Quoted 2 lines · Rakib Mullick

How do you plan to test this power saving scheme? Using powertop? Or, is there any other tools?

We should start out simple enough that we can validate it by looking at task placement by hand, eg. 4 tasks on a dual socket quad-core, should only keep one socket awake.

We can also add an power aware evaluator to Linsched (another one of those things that needs getting sorted).

And yeah, someone running with a power meter is of course king.

_BUT_ we shouldn't go off the wall with power meters as that very quickly gets very specific to the system being measured.

We should really keep this thing as simple as possible while still providing some benefit for all various architectures without tons of per arch knobs and knowhow.

Quoted 2 lines · Matthew Garrett

Are there workloads in which "power" might provide more performance than "performance"? If so, don't use these terms.

Power scheme should no chance has better performance in design.

Quoted 5 lines · Alex Shi

Quoted 2 lines · Matthew Garrett

Are there workloads in which "power" might provide more performance than "performance"? If so, don't use these terms.

Power scheme should no chance has better performance in design.

Power will tend to concentrate processes on packages,

will tend to split them across packages?

processes gain from being on the same package and sharing cache locality?

Quoted 1 line · Matthew Garrett

will tend to split them across packages?

No, there is still has balance idea in this rough proposal. If a domain is not overload, it is better to left old tasks unchanged. I should say, current scheduler is the 'performance' trend scheme.

From what I have understood so far,I try to summarise pin pointed

Quoted 10 lines · Rakib Mullick

Quoted 9 lines · Alex Shi

Since there is no power saving consideration in scheduler CFS, I has a very rough idea for enabling a new power saving schema in CFS.

It bases on the following assumption:
1, If there are many task crowd in system, just let few domain cpus running and let other cpus idle can not save power. Let all cpu take the load, finish tasks early, and then get into idle. will save more power and have better user experience.

Quoted 3 lines

This assumption indirectly point towards the scheme when performance is enabled, isn't it? Cause you're trying to spread the load equally amongst all the CPUs.

It is.

Quoted 4 lines

Quoted 4 lines

select_task_rq_fair()
{

	int powersaving = 0;

Quoted 5 lines

Quoted 5 lines

	for_each_domain(cpu, tmp) {
		if (policy == power && tmp_has_capacity &&
			 tmp->flags & sd_flag) {
			sd = tmp;
			//It is fine to got cpu in the domain

			powersaving = 1;

Quoted 5 lines

Quoted 5 lines

			break;
		}
	}

	while(sd) {

if (policy == power && powersaving == 1)

Quoted 2 lines

Quoted 1 line

find_busiest_and_capable_group()

Quoted 3 lines

I'm not sure what find_busiest_and_capable_group() would really be, it seems it'll find the busiest and capable group, but isn't it a conflict with the first assumption you proposed on your proposal?

This pseudo code missed a power saving workable flag , adding it into above code should solved your concern.

Hi everyone,

From what I have understood so far,I try to summarise pin pointed differences between the performance and power policies as found relevant to the scheduler-load balancing mechanism.Any thoughts?

Hi everyone,

From what I have understood so far,I try to summarise pin pointed differences between the performance and power policies as found relevant to the scheduler-load balancing mechanism.Any thoughts?

*Performance policy*:

Q1.Who triggers load_balance?
Load balance is triggered when a cpu is found to be idle.(Pull mechanism)

Q2.How is load_balance handled?
When triggered,the load is looked to be pulled from its sched domain. First the sched groups in the domain the cpu belongs to is queried followed by the runqueues in the busiest group.then the tasks are moved.

This course of action is found analogous to the performance policy because:

1.First the idle cpu initiates the pull action
2.The busiest cpu hands over the load to this cpu.A person who can handle any work is querying as to who cannot handle more work.

*Power policy*:

So how is power policy different? As Peter says,'pack more than spread more'.

Q1.Who triggers load balance?
It is the cpu which cannot handle more work.Idle cpu is left to remain idle.(Push mechanism)

Q2.How is load_balance handled?
First the least busy runqueue,from within the sched_group that the busy cpu belongs to is queried.if none exist,ie all the runqueues are equally busy then move on to the other sched groups.

Here again the 'least busy' policy should be applied,first at group level then at the runqueue level.

This course of action is found analogous to the power policy because as much as possible busy and capable cpus within a small range try to handle the existing load.

Quoted 5 lines · Matthew Garrett

Quoted 2 lines · Alex Shi

power aware scheduling), this proposal will adopt the sched_balance_policy concept and use 2 kind of policy: performance, power.

Quoted 3 lines

Are there workloads in which "power" might provide more performance than "performance"? If so, don't use these terms.

Power scheme should no chance has better performance in design.

*Power policy*:

So how is power policy different? As Peter says,'pack more than spread more'.

Quoted 11 lines

sub proposal:
1, If it's possible to balance task on idlest cpu not appointed 'balance cpu'. If so, it may can reduce one more time balancing.
The idlest cpu can prefer the new idle cpu; and is the least load cpu; 2, se or task load is good for running time setting.
but it should the second basis in load balancing. The first basis of LB is running tasks' number in group/cpu. Since whatever of the weight of groups is, if the tasks number is less than cpu number, the group is still has capacity to take more tasks. (will consider the SMT cpu power or other big/little cpu capacity on ARM.)

Ah, no we shouldn't balance on nr_running, but on the amount of time consumed. Imagine two tasks being woken at the same time, both tasks will only run a fraction of the available time, you don't want this to exceed your capacity because ran back to back the one cpu will still be mostly idle.

What you want it to keep track of a per-cpu utilization level (inverse of idle-time) and using PJTs per-task runnable avg see if placing the new task on will exceed the utilization limit.

I think some of the Linaro people actually played around with this, Vincent?

Quoted 4 lines

*Power policy*:

So how is power policy different? As Peter says,'pack more than spread more'.

this is ... a dubiously general statement.

for good power, at least on Intel cpus, you want to spread. Parallelism is efficient.

the only thing you do not want to do, is wake cpus up for tasks that only run extremely briefly (think "100 usec" or less).

so maybe the balance interval is slightly different, or more, you don't balance tasks that historically ran only for brief periods

The c-state governor can call the scheduler code before putting a CPU to sleep, to indicate (1) the wakeup latency of the CPU, and (2) whether TLB and/or cache get invalidated.

Quoted 4 lines

*Power policy*:

So how is power policy different? As Peter says,'pack more than spread more'.

this is ... a dubiously general statement.

for good power, at least on Intel cpus, you want to spread. Parallelism is efficient.

the only thing you do not want to do, is wake cpus up for tasks that only run extremely briefly (think "100 usec" or less).

so maybe the balance interval is slightly different, or more, you don't balance tasks that historically ran only for brief periods

Quoted 2 lines · Alex Shi

Since there is no power saving consideration in scheduler CFS, I has a very rough idea for enabling a new power saving schema in CFS.

Adding Thomas, he always delights poking holes in power schemes.

Quoted 5 lines

It bases on the following assumption:
1, If there are many task crowd in system, just let few domain cpus running and let other cpus idle can not save power. Let all cpu take the load, finish tasks early, and then get into idle. will save more power and have better user experience.

I'm not sure this is a valid assumption. I've had it explained to me by various people that race-to-idle isn't always the best thing. It has to do with the cost of switching power states and the duration of execution and other such things.

Quoted 3 lines

2, schedule domain, schedule group perfect match the hardware, and the power consumption unit. So, pull tasks out of a domain means potentially this power consumption unit idle.

I'm not sure I understand what you're saying, sorry.

Quoted 3 lines

So, according Peter mentioned in commit 8e7fbcbc22c(sched: Remove stale power aware scheduling), this proposal will adopt the sched_balance_policy concept and use 2 kind of policy: performance, power.

Yay, ideally we'd also provide a 3rd option: auto, which simply switches between the two based on AC/BAT, UPS status and simple things like that. But this seems like a later concern, you have to have something to pick between before you can pick :-)

Quoted 2 lines

And in scheduling, 2 place will care the policy, load_balance() and in task fork/exec: select_task_rq_fair().

ack

Quoted 2 lines

Here is some pseudo code try to explain the proposal behaviour in load_balance() and select_task_rq_fair();

Oh man.. A few words outlining the general idea would've been nice.

Quoted 8 lines

load_balance() {
      update_sd_lb_stats(); //get busiest group, idlest group data.

      if (sd->nr_running > sd's capacity) {
              //power saving policy is not suitable for
              //this scenario, it runs like performance policy
              mv tasks from busiest cpu in busiest group to
              idlest  cpu in idlest group;

Once upon a time we talked about adding a factor to the capacity for this. So say you'd allow 2*capacity before overflowing and waking another power group.

But I think we should not go on nr_running here, PJTs per-entity load tracking stuff gives us much better measures -- also, repost that series already Paul! :-)

Also, I'm not sure this is entirely correct, the thing you want to do for power aware stuff is to minimize the number of active power domains, this means you don't want idlest, you want least busy non-idle.

Quoted 9 lines

      } else {// the sd has enough capacity to hold all tasks.
              if (sg->nr_running > sg's capacity) {
                      //imbalanced between groups
                      if (schedule policy == performance) {
                              //when 2 busiest group at same busy
                              //degree, need to prefer the one has
                              // softest group??
                              move tasks from busiest group to
                                      idletest group;

So I'd leave the currently implemented scheme as performance, and I don't think the above describes the current state.

Quoted 6 lines

                      } else if (schedule policy == power)
                              move tasks from busiest group to
                              idlest group until busiest is just full
                              of capacity.
                              //the busiest group can balance
                              //internally after next time LB,

There's another thing we need to do, and that is collect tasks in a minimal amount of power domains. The old code (that got deleted) did something like that, you can revive some of the that code if needed -- I just killed everything to be able to start with a clean slate.

Quoted 16 lines

              } else {
                      //all groups has enough capacity for its tasks.
                      if (schedule policy == performance)
                              //all tasks may has enough cpu
                              //resources to run,
                              //mv tasks from busiest to idlest group?
                              //no, at this time, it's better to keep
                              //the task on current cpu.
                              //so, it is maybe better to do balance
                              //in each of groups
                              for_each_imbalance_groups()
                                      move tasks from busiest cpu to
                                      idlest cpu in each of groups;
                      else if (schedule policy == power) {
                              if (no hard pin in idlest group)
                                      mv tasks from idlest group to

Quoted 8 lines

                                      busiest until busiest full.
                              else
                                      mv unpin tasks to the biggest
                                      hard pin group.
                      }
              }
      }
}

OK, so you only start to group later.. I think we can do better than that.

Quoted 11 lines

sub proposal:
1, If it's possible to balance task on idlest cpu not appointed 'balance cpu'. If so, it may can reduce one more time balancing.
The idlest cpu can prefer the new idle cpu; and is the least load cpu; 2, se or task load is good for running time setting.
but it should the second basis in load balancing. The first basis of LB is running tasks' number in group/cpu. Since whatever of the weight of groups is, if the tasks number is less than cpu number, the group is still has capacity to take more tasks. (will consider the SMT cpu power or other big/little cpu capacity on ARM.)

Ah, no we shouldn't balance on nr_running, but on the amount of time consumed. Imagine two tasks being woken at the same time, both tasks will only run a fraction of the available time, you don't want this to exceed your capacity because ran back to back the one cpu will still be mostly idle.

What you want it to keep track of a per-cpu utilization level (inverse of idle-time) and using PJTs per-task runnable avg see if placing the new task on will exceed the utilization limit.

I think some of the Linaro people actually played around with this, Vincent?

Quoted 3 lines

unsolved issues:
1, like current scheduler, it didn't handled cpu affinity well in load_balance.

cpu affinity is always 'fun'.. while there's still a few fun sites in the current load-balancer we do better than we did a while ago.

Quoted 1 line

2, task group that isn't consider well in this rough proposal.

You mean the cgroup mess?

Quoted 2 lines

It isn't consider well and may has mistaken . So just share my ideas and hope it become better and workable in your comments and discussion.

Very simplistically the current scheme is a 'spread' the load scheme (SD_PREFER_SIBLING if you will). We spread load to maximize per-task cache and cpu power.

The power scheme should be a 'pack' scheme, where we minimize the active power domains.

One way to implement this is to keep track of an active and under-utilized power domain (the target) and fail the regular (pull) load-balance for all cpus not in that domain. For the cpu that are in that domain we'll have find_busiest select from all other under-utilized domains pulling tasks to fill our target, once full, we pick a new target, goto 1.

Quoted 13 lines · Matthew Garrett

Quoted 11 lines · Peter Zijlstra

Quoted 5 lines · Alex Shi

It bases on the following assumption:
1, If there are many task crowd in system, just let few domain cpus running and let other cpus idle can not save power. Let all cpu take the load, finish tasks early, and then get into idle. will save more power and have better user experience.

I'm not sure this is a valid assumption. I've had it explained to me by various people that race-to-idle isn't always the best thing. It has to do with the cost of switching power states and the duration of execution and other such things.

Quoted 1 line

This is affected by Intel's implementation - if there's a single active

not just intel.. also AMD
basically everyone who has the memory controller in the cpu package will end up with a restriction very similar to this.

(this is because the exit-from-self-refresh latency is pretty high.. at least in DDR2/3)

Quoted 4 lines

*Power policy*:

So how is power policy different? As Peter says,'pack more than spread more'.

this is ... a dubiously general statement.

for good power, at least on Intel cpus, you want to spread. Parallelism is efficient.

Quoted 5 lines · Arjan van de Ven

Quoted 4 lines

*Power policy*:

So how is power policy different? As Peter says,'pack more than spread more'.

Quoted 3 lines

this is ... a dubiously general statement.

for good power, at least on Intel cpus, you want to spread. Parallelism is efficient.

Is this really true? In a two-socket system I'd have thought the benefit of keeping socket 1 in package C3 outweighed the cost of keeping socket 0 awake for slightly longer.

Quoted 5 lines · Matthew Garrett

Quoted 3 lines · Arjan van de Ven

this is ... a dubiously general statement.

for good power, at least on Intel cpus, you want to spread. Parallelism is efficient.

Quoted 3 lines

Is this really true? In a two-socket system I'd have thought the benefit of keeping socket 1 in package C3 outweighed the cost of keeping socket 0 awake for slightly longer.

not on Intel

you can't enter package c3 either until every one is down. (e.g. memory controller must stay on etc etc)

Quoted 8 lines · Arjan van de Ven

Quoted 7 lines · Matthew Garrett

Quoted 3 lines · Arjan van de Ven

this is ... a dubiously general statement.

for good power, at least on Intel cpus, you want to spread. Parallelism is efficient.

Is this really true? In a two-socket system I'd have thought the benefit of keeping socket 1 in package C3 outweighed the cost of keeping socket 0 awake for slightly longer.

Quoted 4 lines

not on Intel

you can't enter package c3 either until every one is down. (e.g. memory controller must stay on etc etc)

I thought that was only PC6 - is there any reason why the package cache can't be entirely powered down?

On 08/17/2012 12:47 PM, Matthew Garrett wrote:

The datasheet for the Xeon E5 (my variant at least) says it doesn't do C7 so never powers down the LLC. However, as you said earlier once you can put the socket into C6 which saves about 30W compared to C1E.

So as far as I can see with this CPU at least you would benefit from shutting down a whole socket when possible.

Quoted 7 lines · Chris Friesen

On 08/17/2012 12:47 PM, Matthew Garrett wrote:

The datasheet for the Xeon E5 (my variant at least) says it doesn't do C7 so never powers down the LLC. However, as you said earlier once you can put the socket into C6 which saves about 30W compared to C1E.

Quoted 2 lines

So as far as I can see with this CPU at least you would benefit from shutting down a whole socket when possible.

Having any active cores on the system prevents all packages from going into PC6 or deeper. What I'm not clear on is whether less deep package C states are also blocked.

Quoted 11 lines · Matthew Garrett

Quoted 9 lines · Chris Friesen

On 08/17/2012 12:47 PM, Matthew Garrett wrote:

The datasheet for the Xeon E5 (my variant at least) says it doesn't do C7 so never powers down the LLC. However, as you said earlier once you can put the socket into C6 which saves about 30W compared to C1E.

Quoted 8 lines

Quoted 2 lines

So as far as I can see with this CPU at least you would benefit from shutting down a whole socket when possible.

Having any active cores on the system prevents all packages from going into PC6 or deeper. What I'm not clear on is whether less deep package C states are also blocked.

Right, we need the memory controller.

The E5 datasheet is a bit ambiguous, it reads:

A processor enters the package C3 low power state when:
-At least one core is in the C3 state.
-The other cores are in a C3 or lower power state, and the processor has
been granted permission by the platform.

Unfortunately it doesn't specify whether that is the other cores in the package, or the other cores on the whole system.

saving mode. But obviously, we need to spread as much as possible across all cores in another socket(to race to idle). So from the example above, we see a threshold that we need to reference before selecting one from two complete different policy: spread or not spread... As long as there is hardware limitation, we could always need knob like that referenced threshold to adapt on different hardware in one kernel....

Hi all,

Quoted 18 lines · Peter Zijlstra

Quoted 11 lines

sub proposal:
1, If it's possible to balance task on idlest cpu not appointed 'balance cpu'. If so, it may can reduce one more time balancing.
The idlest cpu can prefer the new idle cpu; and is the least load cpu; 2, se or task load is good for running time setting.
but it should the second basis in load balancing. The first basis of LB is running tasks' number in group/cpu. Since whatever of the weight of groups is, if the tasks number is less than cpu number, the group is still has capacity to take more tasks. (will consider the SMT cpu power or other big/little cpu capacity on ARM.)

Ah, no we shouldn't balance on nr_running, but on the amount of time consumed. Imagine two tasks being woken at the same time, both tasks will only run a fraction of the available time, you don't want this to exceed your capacity because ran back to back the one cpu will still be mostly idle.

Quoted 7 lines

What you want it to keep track of a per-cpu utilization level (inverse of idle-time) and using PJTs per-task runnable avg see if placing the new task on will exceed the utilization limit.

I think some of the Linaro people actually played around with this, Vincent?

I agree. A better measure of cpu load and task weight than nr_running and the current task load weight are necessary to do proper task packing.

I have used PJTs per-task load-tracking for scheduling experiments on heterogeneous systems and my experience is that it works quite well for determining the load of a specific task. Something like PJTs work would be a good starting point for power aware scheduling and better support for heterogeneous systems.

One of the biggest challenges here for load-balancing is translating task load from one cpu to another as the task load is influenced by the total load of its cpu. So a task that appears to be heavy on an oversubscribed cpu might not be so heavy after all when it is moved to a cpu with plenty cpu time to spare. This issue is likely to be more pronounced on heterogeneous systems and system with aggressive frequency scaling. It might be possible to avoid having to translate load or that it doesn't really matter, but I haven't completely convinced myself yet.

My point is that getting the task load right or at least better is a fundamental requirement for improving power aware scheduling.

Quoted 9 lines · Peter Zijlstra

This all sounds far too complicated.. we're talking about simple spreading and packing balancers without deep arch knowledge and knobs, we couldn't possibly evaluate anything like that.

I was really more thinking of something useful for the laptops out there, when they pull the power cord it makes sense to try and keep CPUs asleep until the one that's awake is saturated.

as long as you don't do that on machines with an Intel CPU.. since that'd be the worst case behavior for tasks that run for more than 100 usec. (e.g. not interrupts, but almost everything else)

Quoted 4 lines

Quoted 4 lines

I was really more thinking of something useful for the laptops out there, when they pull the power cord it makes sense to try and keep CPUs asleep until the one that's awake is saturated.

s/CPU/core ?

There's also cases where the kernel has insufficient information from the hardware and from the admin about the preferred characteristics/policy of the system - a tweakable fallback knob might be provided for that sad case.

The point is, that knob is not the policy setting and it's not the main mechanism. It's a fallback.

Quoted 2 lines · Alex Shi

Since there is no power saving consideration in scheduler CFS, I has a very rough idea for enabling a new power saving schema in CFS.

Adding Thomas, he always delights poking holes in power schemes.

Quoted 5 lines

It bases on the following assumption:
1, If there are many task crowd in system, just let few domain cpus running and let other cpus idle can not save power. Let all cpu take the load, finish tasks early, and then get into idle. will save more power and have better user experience.

I'm not sure this is a valid assumption. I've had it explained to me by various people that race-to-idle isn't always the best thing. It has to do with the cost of switching power states and the duration of execution and other such things.

Quoted 3 lines

2, schedule domain, schedule group perfect match the hardware, and the power consumption unit. So, pull tasks out of a domain means potentially this power consumption unit idle.

I'm not sure I understand what you're saying, sorry.

Quoted 3 lines

So, according Peter mentioned in commit 8e7fbcbc22c(sched: Remove stale power aware scheduling), this proposal will adopt the sched_balance_policy concept and use 2 kind of policy: performance, power.

Yay, ideally we'd also provide a 3rd option: auto, which simply switches between the two based on AC/BAT, UPS status and simple things like that. But this seems like a later concern, you have to have something to pick between before you can pick :-)

Quoted 2 lines

And in scheduling, 2 place will care the policy, load_balance() and in task fork/exec: select_task_rq_fair().

ack

Quoted 2 lines

Here is some pseudo code try to explain the proposal behaviour in load_balance() and select_task_rq_fair();

Oh man.. A few words outlining the general idea would've been nice.

Quoted 8 lines

load_balance() {
      update_sd_lb_stats(); //get busiest group, idlest group data.

      if (sd->nr_running > sd's capacity) {
              //power saving policy is not suitable for
              //this scenario, it runs like performance policy
              mv tasks from busiest cpu in busiest group to
              idlest  cpu in idlest group;

Once upon a time we talked about adding a factor to the capacity for this. So say you'd allow 2*capacity before overflowing and waking another power group.

But I think we should not go on nr_running here, PJTs per-entity load tracking stuff gives us much better measures -- also, repost that series already Paul! :-)

Also, I'm not sure this is entirely correct, the thing you want to do for power aware stuff is to minimize the number of active power domains, this means you don't want idlest, you want least busy non-idle.

Quoted 9 lines

      } else {// the sd has enough capacity to hold all tasks.
              if (sg->nr_running > sg's capacity) {
                      //imbalanced between groups
                      if (schedule policy == performance) {
                              //when 2 busiest group at same busy
                              //degree, need to prefer the one has
                              // softest group??
                              move tasks from busiest group to
                                      idletest group;

So I'd leave the currently implemented scheme as performance, and I don't think the above describes the current state.

Quoted 6 lines

                      } else if (schedule policy == power)
                              move tasks from busiest group to
                              idlest group until busiest is just full
                              of capacity.
                              //the busiest group can balance
                              //internally after next time LB,

There's another thing we need to do, and that is collect tasks in a minimal amount of power domains. The old code (that got deleted) did something like that, you can revive some of the that code if needed -- I just killed everything to be able to start with a clean slate.

Quoted 16 lines

              } else {
                      //all groups has enough capacity for its tasks.
                      if (schedule policy == performance)
                              //all tasks may has enough cpu
                              //resources to run,
                              //mv tasks from busiest to idlest group?
                              //no, at this time, it's better to keep
                              //the task on current cpu.
                              //so, it is maybe better to do balance
                              //in each of groups
                              for_each_imbalance_groups()
                                      move tasks from busiest cpu to
                                      idlest cpu in each of groups;
                      else if (schedule policy == power) {
                              if (no hard pin in idlest group)
                                      mv tasks from idlest group to

Quoted 8 lines

                                      busiest until busiest full.
                              else
                                      mv unpin tasks to the biggest
                                      hard pin group.
                      }
              }
      }
}

OK, so you only start to group later.. I think we can do better than that.

Quoted 11 lines

sub proposal:
1, If it's possible to balance task on idlest cpu not appointed 'balance cpu'. If so, it may can reduce one more time balancing.
The idlest cpu can prefer the new idle cpu; and is the least load cpu; 2, se or task load is good for running time setting.
but it should the second basis in load balancing. The first basis of LB is running tasks' number in group/cpu. Since whatever of the weight of groups is, if the tasks number is less than cpu number, the group is still has capacity to take more tasks. (will consider the SMT cpu power or other big/little cpu capacity on ARM.)

Ah, no we shouldn't balance on nr_running, but on the amount of time consumed. Imagine two tasks being woken at the same time, both tasks will only run a fraction of the available time, you don't want this to exceed your capacity because ran back to back the one cpu will still be mostly idle.

What you want it to keep track of a per-cpu utilization level (inverse of idle-time) and using PJTs per-task runnable avg see if placing the new task on will exceed the utilization limit.

I think some of the Linaro people actually played around with this, Vincent?

Quoted 4 lines

unsolved issues:
1, like current scheduler, it didn't handled cpu affinity well in load_balance.

cpu affinity is always 'fun'.. while there's still a few fun sites in the current load-balancer we do better than we did a while ago.

Quoted 1 line

2, task group that isn't consider well in this rough proposal.

You mean the cgroup mess?

Quoted 2 lines

It isn't consider well and may has mistaken . So just share my ideas and hope it become better and workable in your comments and discussion.

Very simplistically the current scheme is a 'spread' the load scheme (SD_PREFER_SIBLING if you will). We spread load to maximize per-task cache and cpu power.

The power scheme should be a 'pack' scheme, where we minimize the active power domains.

One way to implement this is to keep track of an active and under-utilized power domain (the target) and fail the regular (pull) load-balance for all cpus not in that domain. For the cpu that are in that domain we'll have find_busiest select from all other under-utilized domains pulling tasks to fill our target, once full, we pick a new target, goto 1.

Quoted 5 lines · Matthew Garrett

Quoted 2 lines · Alex Shi

power aware scheduling), this proposal will adopt the sched_balance_policy concept and use 2 kind of policy: performance, power.

Quoted 3 lines

Are there workloads in which "power" might provide more performance than "performance"? If so, don't use these terms.

Power scheme should no chance has better performance in design.

So please make sure that there are obvious and easy ways to switch this stuff off or provide "low latency" know that keeps the system from assuming that idle time means that full performance is not needed.

Quoted 51 lines · Peter Zijlstra

Quoted 2 lines · Alex Shi

Since there is no power saving consideration in scheduler CFS, I has a very rough idea for enabling a new power saving schema in CFS.

Adding Thomas, he always delights poking holes in power schemes.

Quoted 5 lines

It bases on the following assumption:
1, If there are many task crowd in system, just let few domain cpus running and let other cpus idle can not save power. Let all cpu take the load, finish tasks early, and then get into idle. will save more power and have better user experience.

I'm not sure this is a valid assumption. I've had it explained to me by various people that race-to-idle isn't always the best thing. It has to do with the cost of switching power states and the duration of execution and other such things.

Quoted 3 lines

2, schedule domain, schedule group perfect match the hardware, and the power consumption unit. So, pull tasks out of a domain means potentially this power consumption unit idle.

I'm not sure I understand what you're saying, sorry.

Quoted 3 lines

So, according Peter mentioned in commit 8e7fbcbc22c(sched: Remove stale power aware scheduling), this proposal will adopt the sched_balance_policy concept and use 2 kind of policy: performance, power.

Yay, ideally we'd also provide a 3rd option: auto, which simply switches between the two based on AC/BAT, UPS status and simple things like that. But this seems like a later concern, you have to have something to pick between before you can pick :-)

Quoted 2 lines

And in scheduling, 2 place will care the policy, load_balance() and in task fork/exec: select_task_rq_fair().

ack

Quoted 2 lines

Here is some pseudo code try to explain the proposal behaviour in load_balance() and select_task_rq_fair();

Oh man.. A few words outlining the general idea would've been nice.

Quoted 8 lines

load_balance() {
      update_sd_lb_stats(); //get busiest group, idlest group data.

      if (sd->nr_running > sd's capacity) {
              //power saving policy is not suitable for
              //this scenario, it runs like performance policy
              mv tasks from busiest cpu in busiest group to
              idlest  cpu in idlest group;

Quoted 7 lines

Once upon a time we talked about adding a factor to the capacity for this. So say you'd allow 2*capacity before overflowing and waking another power group.

But I think we should not go on nr_running here, PJTs per-entity load tracking stuff gives us much better measures -- also, repost that series already Paul! :-)

Yes -- I just got back from Africa this week. It's updated for almost all the previous comments but I ran out of time before I left to re-post. I'm just about caught up enough that I should be able to get this done over the upcoming weekend. Monday at the latest.

Quoted 77 lines

Also, I'm not sure this is entirely correct, the thing you want to do for power aware stuff is to minimize the number of active power domains, this means you don't want idlest, you want least busy non-idle.

Quoted 9 lines

      } else {// the sd has enough capacity to hold all tasks.
              if (sg->nr_running > sg's capacity) {
                      //imbalanced between groups
                      if (schedule policy == performance) {
                              //when 2 busiest group at same busy
                              //degree, need to prefer the one has
                              // softest group??
                              move tasks from busiest group to
                                      idletest group;

So I'd leave the currently implemented scheme as performance, and I don't think the above describes the current state.

Quoted 6 lines

                      } else if (schedule policy == power)
                              move tasks from busiest group to
                              idlest group until busiest is just full
                              of capacity.
                              //the busiest group can balance
                              //internally after next time LB,

There's another thing we need to do, and that is collect tasks in a minimal amount of power domains. The old code (that got deleted) did something like that, you can revive some of the that code if needed -- I just killed everything to be able to start with a clean slate.

Quoted 16 lines

              } else {
                      //all groups has enough capacity for its tasks.
                      if (schedule policy == performance)
                              //all tasks may has enough cpu
                              //resources to run,
                              //mv tasks from busiest to idlest group?
                              //no, at this time, it's better to keep
                              //the task on current cpu.
                              //so, it is maybe better to do balance
                              //in each of groups
                              for_each_imbalance_groups()
                                      move tasks from busiest cpu to
                                      idlest cpu in each of groups;
                      else if (schedule policy == power) {
                              if (no hard pin in idlest group)
                                      mv tasks from idlest group to

Quoted 8 lines

                                      busiest until busiest full.
                              else
                                      mv unpin tasks to the biggest
                                      hard pin group.
                      }
              }
      }
}

OK, so you only start to group later.. I think we can do better than that.

Quoted 11 lines

sub proposal:
1, If it's possible to balance task on idlest cpu not appointed 'balance cpu'. If so, it may can reduce one more time balancing.
The idlest cpu can prefer the new idle cpu; and is the least load cpu; 2, se or task load is good for running time setting.
but it should the second basis in load balancing. The first basis of LB is running tasks' number in group/cpu. Since whatever of the weight of groups is, if the tasks number is less than cpu number, the group is still has capacity to take more tasks. (will consider the SMT cpu power or other big/little cpu capacity on ARM.)

Ah, no we shouldn't balance on nr_running, but on the amount of time consumed. Imagine two tasks being woken at the same time, both tasks will only run a fraction of the available time, you don't want this to exceed your capacity because ran back to back the one cpu will still be mostly idle.

Quoted 3 lines

What you want it to keep track of a per-cpu utilization level (inverse of idle-time) and using PJTs per-task runnable avg see if placing the new task on will exceed the utilization limit.

Observations of the runnable average also have the nice property that it quickly converges to 100% when over-scheduled.

Since we also have the usage average for a single task the ratio of used avg:runnable avg is likely a useful pointwise estimate.

Quoted 34 lines

I think some of the Linaro people actually played around with this, Vincent?

Quoted 3 lines

unsolved issues:
1, like current scheduler, it didn't handled cpu affinity well in load_balance.

cpu affinity is always 'fun'.. while there's still a few fun sites in the current load-balancer we do better than we did a while ago.

Quoted 1 line

2, task group that isn't consider well in this rough proposal.

You mean the cgroup mess?

Quoted 2 lines

It isn't consider well and may has mistaken . So just share my ideas and hope it become better and workable in your comments and discussion.

Very simplistically the current scheme is a 'spread' the load scheme (SD_PREFER_SIBLING if you will). We spread load to maximize per-task cache and cpu power.

The power scheme should be a 'pack' scheme, where we minimize the active power domains.

One way to implement this is to keep track of an active and under-utilized power domain (the target) and fail the regular (pull) load-balance for all cpus not in that domain. For the cpu that are in that domain we'll have find_busiest select from all other under-utilized domains pulling tasks to fill our target, once full, we pick a new target, goto 1.

If the answer is 'yes' then there's clear cases where the kernel (should) automatically know the events where we switch from balancing for performance to balancing for power:

Quoted 3 lines · Christoph Lameter

So please make sure that there are obvious and easy ways to switch this stuff off or provide "low latency" know that keeps the system from assuming that idle time means that full performance is not needed.

That seems like an issue for cpuidle, not the scheduler. Does pm_qos not already do what you want?

Quoted 6 lines

Quoted 6 lines

What you want it to keep track of a per-cpu utilization level (inverse of idle-time) and using PJTs per-task runnable avg see if placing the new task on will exceed the utilization limit.

I think some of the Linaro people actually played around with this, Vincent?

Sorry for the late reply but I had almost no network access during last weeks.

So Linaro also works on a power aware scheduler as Peter mentioned.

Based on previous tests, we have concluded that main drawback of the (now removed) old power scheduler was that we had no way to make difference between short and long running tasks whereas it's a key input (at least for phone) for deciding to pack tasks and for selecting the core on an asymmetric system.

One additional key information is the power distribution in the system which can have a finer granularity than current sched_domain description. Peter's proposal was to use a SHARE_POWERLINE flag similarly to flags that already describe if a sched_domain share resources or cpu capacity.

With these 2 new information, we can have a 1st power saving scheduler which spread or packed tasks across core and package

Vincent

Quoted 12 lines · Alex Shi

Quoted 8 lines · Matthew Garrett

Quoted 2 lines · Alex Shi

power aware scheduling), this proposal will adopt the sched_balance_policy concept and use 2 kind of policy: performance, power.

Are there workloads in which "power" might provide more performance than "performance"? If so, don't use these terms.

Quoted 1 line

Power scheme should no chance has better performance in design.

A side effect of packing small tasks on one core is that you always use the core with the lowest C-state which will minimize the wake up latency so you can sometime get better results than performance mode which will try to use a other core in another cluster which will take more time to wake up that waiting for the end of the current task.

Quoted 3 lines · Ingo Molnar

If the answer is 'yes' then there's clear cases where the kernel (should) automatically know the events where we switch from balancing for performance to balancing for power:

No. We can't identify all of these cases and we can't identify corner cases. [...]

[...] Putting this kind of policy in the kernel is an awful idea. [...]

[...] It should never be altering policy itself, [...]

[...] because it'll get it wrong and people will file bugs complaining that it got it wrong and the biggest case where you *need* to be able to handle switching between performance and power optimisations (your rack management unit just told you that you're going to have to drop power consumption by 20W) is one where the kernel doesn't have all the information it needs to do this. So why bother at all?

Quoted 8 lines · Vincent Guittot

Quoted 6 lines

Quoted 6 lines

What you want it to keep track of a per-cpu utilization level (inverse of idle-time) and using PJTs per-task runnable avg see if placing the new task on will exceed the utilization limit.

I think some of the Linaro people actually played around with this, Vincent?

Sorry for the late reply but I had almost no network access during last weeks.

Quoted 7 lines

So Linaro also works on a power aware scheduler as Peter mentioned.

Based on previous tests, we have concluded that main drawback of the (now removed) old power scheduler was that we had no way to make difference between short and long running tasks whereas it's a key input (at least for phone) for deciding to pack tasks and for selecting the core on an asymmetric system.

It is hard to estimate future in general view point. but from hack point, maybe you can add something to hint this from task_struct. :)

Quoted 5 lines

One additional key information is the power distribution in the system which can have a finer granularity than current sched_domain description. Peter's proposal was to use a SHARE_POWERLINE flag similarly to flags that already describe if a sched_domain share resources or cpu capacity.

Seems I missed this. what's difference with current SD_SHARE_CPUPOWER and SD_SHARE_PKG_RESOURCES.

Quoted 4 lines

With these 2 new information, we can have a 1st power saving scheduler which spread or packed tasks across core and package

Fine, I like to test them on X86, plus SMT and NUMA :)

Quoted 2 lines

Vincent

Quoted 2 lines · Matthew Garrett

[...] Putting this kind of policy in the kernel is an awful idea. [...]

A modern kernel better know what state the system is in: on battery or on AC power.

Quoted 1 line

[...] It should never be altering policy itself, [...]

The kernel/scheduler simply offers sensible defaults where it can. User-space can augment/modify/override that in any which way it wishes to.

This stuff has not been properly sorted out in the last 10+ years since we have battery driven devices, so we might as well start with the kernel offering sane default behavior where it can ...

Quoted 7 lines

[...] because it'll get it wrong and people will file bugs complaining that it got it wrong and the biggest case where you *need* to be able to handle switching between performance and power optimisations (your rack management unit just told you that you're going to have to drop power consumption by 20W) is one where the kernel doesn't have all the information it needs to do this. So why bother at all?

The point is to have a working default mechanism.

Quoted 6 lines · Ingo Molnar

Quoted 2 lines · Matthew Garrett

[...] Putting this kind of policy in the kernel is an awful idea. [...]

A modern kernel better know what state the system is in: on battery or on AC power.

That's a fundamentally uninteresting thing for the kernel to know about. [...]

[...] AC/battery is just not an important power management policy input when compared to various other things.

Quoted 6 lines

Quoted 1 line

[...] It should never be altering policy itself, [...]

The kernel/scheduler simply offers sensible defaults where it can. User-space can augment/modify/override that in any which way it wishes to.

Quoted 4 lines

This stuff has not been properly sorted out in the last 10+ years since we have battery driven devices, so we might as well start with the kernel offering sane default behavior where it can ...

Userspace has been doing a perfectly reasonable job of determining policy here.

Quoted 9 lines

Quoted 8 lines

[...] because it'll get it wrong and people will file bugs complaining that it got it wrong and the biggest case where you *need* to be able to handle switching between performance and power optimisations (your rack management unit just told you that you're going to have to drop power consumption by 20W) is one where the kernel doesn't have all the information it needs to do this. So why bother at all?

Quoted 1 line

The point is to have a working default mechanism.

Your suggestions aren't a working default mechanism.

Quoted 6 lines

Quoted 3 lines

A modern kernel better know what state the system is in: on battery or on AC power.

That's a fundamentally uninteresting thing for the kernel to know about. [...]

I disagree.

Quoted 3 lines

[...] AC/battery is just not an important power management policy input when compared to various other things.

Such as?

The thing is, when I use Linux on a laptop then AC/battery is *the* main policy input.

Quoted 2 lines · Matthew Garrett

[...] AC/battery is just not an important power management policy input when compared to various other things.

Such as?

The thing is, when I use Linux on a laptop then AC/battery is *the* main policy input.

Quoted 2 lines

Userspace has been doing a perfectly reasonable job of determining policy here.

Has it properly switched the scheduler's balancing between power-effient and performance-maximizing strategies when for example a laptop's AC got unplugged/replugged?

Quoted 2 lines

That's a fundamentally uninteresting thing for the kernel to know about. [...]

I disagree.

Quoted 2 lines

[...] AC/battery is just not an important power management policy input when compared to various other things.

Such as?

The thing is, when I use Linux on a laptop then AC/battery is *the* main policy input.

Quoted 2 lines

Userspace has been doing a perfectly reasonable job of determining policy here.

Has it properly switched the scheduler's balancing between power-effient and performance-maximizing strategies when for example a laptop's AC got unplugged/replugged?

Quoted 1 line

Your suggestions aren't a working default mechanism.

In what way?

Quoted 5 lines · Ingo Molnar

Quoted 2 lines · Matthew Garrett

[...] AC/battery is just not an important power management policy input when compared to various other things.

Such as?

The scheduler's behaviour is going to have a minimal impact on power consumption on laptops. Other things are much more important - backlight level, ASPM state, that kind of thing. So why special case the scheduler? [...]

[...] This is going to be hugely more important on multi-socket systems, where your policy is usually going to be dictated by the specific workload that you're running at the time. [...]

[...] The exception is in cases where your rack is overcommitted for power and your rack management unit is telling you to reduce power consumption since otherwise it's going to have to cut the power to one of the machines in the rack in the next few seconds.

Quoted 2 lines

The thing is, when I use Linux on a laptop then AC/battery is *the* main policy input.

And it's already well handled from userspace, as it has to be.

Quoted 6 lines

Quoted 2 lines

Userspace has been doing a perfectly reasonable job of determining policy here.

Has it properly switched the scheduler's balancing between power-effient and performance-maximizing strategies when for example a laptop's AC got unplugged/replugged?

No, because sched_mt_powersave usually crippled performance more than it saved power and nobody makes multi-socket laptops.

Quoted 4 lines · Matthew Garrett

The scheduler's behaviour is going to have a minimal impact on power consumption on laptops. Other things are much more important - backlight level, ASPM state, that kind of thing. So why special case the scheduler? [...]

I'm not special casing the scheduler - but we are talking about scheduler policies here, so *if* it makes sense to handle this dynamically then obviously the scheduler wants to use system state information when/if the kernel can get it.

Your argument is as if you said that the shape of a car's side view mirrors is not important to its top speed, because the overall shape of the chassis and engine power are much more important.

But we are desiging side view mirrors here, so we might as well do a good job there.

Quoted 4 lines

[...] This is going to be hugely more important on multi-socket systems, where your policy is usually going to be dictated by the specific workload that you're running at the time. [...]

If only we had some kernel subsystem that is intimiately familar with the workloads running on the system and could act accordingly and with low latency.

We could name that subsystem it in some intuitive fashion: it switches and schedules workloads, so how about calling it the 'scheduler'?

Quoted 5 lines

[...] The exception is in cases where your rack is overcommitted for power and your rack management unit is telling you to reduce power consumption since otherwise it's going to have to cut the power to one of the machines in the rack in the next few seconds.

( That must be some ACPI middleware driven crap, right? Not
really the Linux kernel's problem. )

Quoted 4 lines

Quoted 2 lines

The thing is, when I use Linux on a laptop then AC/battery is *the* main policy input.

And it's already well handled from userspace, as it has to be.

Not according to the developers switching away from Linux desktop distros in droves, because MacOSX or Win7 has 30%+ better battery efficiency.

The scheduler might be a small part of the picture, but it's certainly a part of it.

Quoted 3 lines

No, because sched_mt_powersave usually crippled performance more than it saved power and nobody makes multi-socket laptops.

That's a user-space policy management fail right there: why wasn't this fixed? If the default policy is in the kernel we can at least fix it in one place for the most common cases. If it's spread out amongst multiple projects then progress only happens at glacial speed ...

Quoted 16 lines · Ingo Molnar

Quoted 4 lines · Matthew Garrett

The scheduler's behaviour is going to have a minimal impact on power consumption on laptops. Other things are much more important - backlight level, ASPM state, that kind of thing. So why special case the scheduler? [...]

I'm not special casing the scheduler - but we are talking about scheduler policies here, so *if* it makes sense to handle this dynamically then obviously the scheduler wants to use system state information when/if the kernel can get it.

Your argument is as if you said that the shape of a car's side view mirrors is not important to its top speed, because the overall shape of the chassis and engine power are much more important.

Quoted 2 lines

But we are desiging side view mirrors here, so we might as well do a good job there.

If the kernel is going to make power choices automatically then it should do it everywhere, not piecemeal.

The scheduler is unaware of whether I care about a process finishing quickly or whether I care about it consuming less power.

Quoted 6 lines

Quoted 5 lines

Quoted 2 lines

The thing is, when I use Linux on a laptop then
AC/battery is *the* main policy input.

And it's already well handled from userspace, as it has to be.

Quoted 3 lines

Not according to the developers switching away from Linux desktop distros in droves, because MacOSX or Win7 has 30%+ better battery efficiency.

Ok so what you're actually telling me here is that you don't understand anything about power management and where our problems are.

Quoted 2 lines

The scheduler might be a small part of the picture, but it's certainly a part of it.

It's in the drivers, which is where it has been since we went tickless.

Quoted 9 lines

Quoted 3 lines

No, because sched_mt_powersave usually crippled performance more than it saved power and nobody makes multi-socket laptops.

That's a user-space policy management fail right there: why wasn't this fixed? If the default policy is in the kernel we can at least fix it in one place for the most common cases. If it's spread out amongst multiple projects then progress only happens at glacial speed ...

sched_mt_powersave was inherently going to have a huge impact on performance, and with modern chips that would result in the platform consuming more power. It was a feature that was useful for a small number of generations of desktop CPUs - I don't think it would ever skew the power/performance ratio in a useful direction on mobile hardware. But feel free to blame userspace for hardware design.

Quoted 3 lines · Matthew Garrett

The scheduler is unaware of whether I care about a process finishing quickly or whether I care about it consuming less power.

You are posing them as if the two were mutually exclusive, while in reality they are not necessarily exclusive: it's quite possible that the highest (non-turbo) CPU frequency happens to be the most energy efficient one for a CPU with a particular workload ...

You also missed the bit of my mail where I suggested that such user preferences and tolerances can be communicated to the scheduler via a policy toggle - which the scheduler would take into account.

Quoted 3 lines

Ok so what you're actually telling me here is that you don't understand anything about power management and where our problems are.

Huh? In practice we suck today in terms of energy efficiency. That covers both scheduling and other areas.

Saying this out aloud does not tell anything about my understanding of power management...

So please outline a technical point.

You mean the code is in drivers? Or the problem is in drivers?

Quoted 7 lines

sched_mt_powersave was inherently going to have a huge impact on performance, and with modern chips that would result in the platform consuming more power. It was a feature that was useful for a small number of generations of desktop CPUs - I don't think it would ever skew the power/performance ratio in a useful direction on mobile hardware. But feel free to blame userspace for hardware design.

FYI, sched_mt_powersave is *GONE* in recent kernels, because it basically never worked. This thread is about designing and implementing something that actually works.

Why? Good scheduling is useful even in isolation.

I'd like to see actual numbers and evidence on a wide range of workloads the spread/don't spread thing is even measurable given that you've also got to factor in effects like completing faster and turning everything off. I'd *really* like to see such evidence on a laptop,which is your one cited case it might work.

Quoted 1 line

Why? Good scheduling is useful even in isolation.

For power - I suspect it's damn near irrelevant except on a big big machine.

Unless you've sorted out your SATA, fixed your phy handling, optimised your desktop for wakeups and worked down the big wakeup causes one by one it's turd polishing.

PM means fixing the stack top to bottom, and its a whackamole game, each one you fix you find the next. You have to sort the entire stack from desktop apps to kernel.

However benchmarks talk - so lets have some benchmarks ... on a laptop.

[...]

Our power consumption is worse than under other operating systems is almost entirely because only one of our three GPU drivers implements any kind of useful power management. [...]

With deep enough C states it's rather relevant whether we continue to burn +50W for a couple of more milliseconds or not, and whether we have the right information from the scheduler and timer subsystem about how long the next idle period is expected to be and how bursty a given task is.

Quoted 3 lines

PM means fixing the stack top to bottom, and its a whackamole game, each one you fix you find the next. You have to sort the entire stack from desktop apps to kernel.

Moving 'policy' into user-space has been an utter failure, mostly because there's not a single project/subsystem responsible for getting a good result to users. This is why I resist "policy should not be in the kernel" meme here.

Quoted 5 lines

With deep enough C states it's rather relevant whether we continue to burn +50W for a couple of more milliseconds or not, and whether we have the right information from the scheduler and timer subsystem about how long the next idle period is expected to be and how bursty a given task is.

50W for 2mS here and there is an irrelevance compared with burning a continual half a watt due to the upstream tree lack some of the SATA power patches for example.

It's the classic "standby mode" problem - energy efficiency has time as a factor and there are a lot of milliseconds in 5 hours. That means anything continually on rapidly dominates the problem space.

Quoted 4 lines

Quoted 3 lines

PM means fixing the stack top to bottom, and its a whackamole game, each one you fix you find the next. You have to sort the entire stack from desktop apps to kernel.

Quoted 4 lines

Moving 'policy' into user-space has been an utter failure, mostly because there's not a single project/subsystem responsible for getting a good result to users. This is why I resist "policy should not be in the kernel" meme here.

You *can't* fix PM in one place. [...]

[...] Power management is a top to bottom thing. It starts in the hardware and propogates right to the top of the user space stack.

A single stupid behaviour in a desktop app is all it needs to knock the odd hour or two off your battery life. Something is mundane as refreshing a bit of the display all the time keeping the GPU and CPU from sleeping well.

Most distros haven't managed to do power management properly because it is this entire integration problem. Every single piece of the puzzle has to be in place before you get any serious gain.

It's not a kernel v user thing. The kernel can't fix it, random bits of userspace can't fix it. This is effectively a "product level" integration problem.

Quoted 2 lines · Matthew Garrett

[...]

Quoted 3 lines

Our power consumption is worse than under other operating systems is almost entirely because only one of our three GPU drivers implements any kind of useful power management. [...]

... and because our CPU frequency and C state selection logic is doing pretty much the worst possible decisions (on x86 at least).

Regardless, you cannot possibly seriously suggest that because there's even greater suckage elsewhere for some workloads we should not even bother with improving the situation here.

It can be more than an irrelevance if the CPU is saturated - say a game running on a mobile device very commonly saturates the CPU. A third of the energy is spent in the CPU, sometimes more.

Quoted 1 line

You *can't* fix PM in one place. [...]

Preferably one project, not one place - but at least don't go down the false path of:

" Policy always belongs into user-space so the kernel can
continue to do a shitty job even for pieces it could understand better ..."

My opinion is that it depends, and I also think that we are so bad currently (on x86) that we can do little harm by trying to do things better.

Quoted 4 lines

[...] Power management is a top to bottom thing. It starts in the hardware and propogates right to the top of the user space stack.

Partly because it's misdesigned: in practice there's very little true user policy about power saving:

- On mobile devices I almost never tweak policy as a user - sometimes I override screen brightness but that's all (and it's trivial compared to all the many other things that go on).

it should all work. There arent millions of people out there wanting to tweak the heck out of PM.

People prefer no knobs at all - they want good defaults and they want at most a single, intuitive, actionable control to override the automation in 1% of the usecases, such as screen brightness.

Quoted 4 lines

A single stupid behaviour in a desktop app is all it needs to knock the odd hour or two off your battery life. Something is mundane as refreshing a bit of the display all the time keeping the GPU and CPU from sleeping well.

Even with highly powertop-optimized systems that have no such app and have very low wakeup rates we still lag behind the competition.

So why not move most pieces into one well-informed code domain (the kernel) and only expose high level controls, instead of expecting user-space to get it all right.

Then the 'only' job of user-space would be to not be silly when implementing their functionality. (and there's nothing intimately PM about that.)

Kernel design decisions *matter*:

Look for example how moving X lowlevel drivers from user-space into kernel-space enabled GPU level power management to begin with. With the old X method it was essentially impossible. Now it's at least possible.

Or look at how Android adding a high-level interface like suspend blockers materially improved the power saving situation for them.

This learned helplessness that "the kernel can do nothing about PM" is somewhat annoying :-)

Quoted 5 lines · Alan Cox

I'd like to see actual numbers and evidence on a wide range of workloads the spread/don't spread thing is even measurable given that you've also got to factor in effects like completing faster and turning everything off. I'd *really* like to see such evidence on a laptop,which is your one cited case it might work.

For my dinky dual core laptop, I suspect you're right, but for a more powerful laptop, I'd expect spread/don't to be noticeable.

Yeah, hard numbers would be nice to see.

If I had a powerful laptop, I'd kill irq balancing, and all but periodic load balancing, and expect to see a positive result.

Quoted 8 lines · Mike Galbraith

Quoted 5 lines · Alan Cox

I'd like to see actual numbers and evidence on a wide range of workloads the spread/don't spread thing is even measurable given that you've also got to factor in effects like completing faster and turning everything off. I'd *really* like to see such evidence on a laptop,which is your one cited case it might work.

Quoted 2 lines

For my dinky dual core laptop, I suspect you're right, but for a more powerful laptop, I'd expect spread/don't to be noticeable.

yeah if you don't spread, you will waste some power. but.. current linux behavior is to spread. so we can only make it worse.

Quoted 4 lines

Yeah, hard numbers would be nice to see.

If I had a powerful laptop, I'd kill irq balancing, and all but periodic load balancing, and expect to see a positive result.

I'd expect to see a negative result ;-)

Quoted 2 lines · Mike Galbraith

For my dinky dual core laptop, I suspect you're right, but for a more powerful laptop, I'd expect spread/don't to be noticeable.

yeah if you don't spread, you will waste some power. but.. current linux behavior is to spread. so we can only make it worse.

Quoted 7 lines · Arjan van de Ven

Quoted 2 lines · Mike Galbraith

For my dinky dual core laptop, I suspect you're right, but for a more powerful laptop, I'd expect spread/don't to be noticeable.

yeah if you don't spread, you will waste some power. but.. current linux behavior is to spread. so we can only make it worse.

Right. For a single socket system the only thing you can do is use two threads in preference to using two cores. That'll keep an extra core in a deep C state for longer, at the cost of keeping the package out of a deep C state for longer. There might be a win if the two processes benefit from improved L1 cache locality, or if you're talking about

Quoted 5 lines · Matthew Garrett

[...]

Our power consumption is worse than under other operating systems is almost entirely because only one of our three GPU drivers implements any kind of useful power management. [...]

... and because our CPU frequency and C state selection logic is doing pretty much the worst possible decisions (on x86 at least).

Regardless, you cannot possibly seriously suggest that because there's even greater suckage elsewhere for some workloads we should not even bother with improving the situation here.

Anyway, I agree with Alan that actual numbers matter.

Thanks,

Ingo