Customize sched domain via cpuset

Using cpuset, now we can partition the system into multiple sched domains. Then, how about providing different characteristics for each domains?

Quoted 2 lines · Hidetoshi Seto

Using cpuset, now we can partition the system into multiple sched domains. Then, how about providing different characteristics for each domains?

Did you actually see much improvement in any relevant workload from tweaking these parameters? If yes what did you change? And how much did it gain?

Ideally the kernel should perform well without much tweaking out of the box, simply because most users won't tweak. Adding a lot of such parameters would imply giving up on good defaults which is not a good thing.

From what I understand they need very aggressive idle balancing; much

Quoted 13 lines · Andi Kleen

Quoted 2 lines · Hidetoshi Seto

Using cpuset, now we can partition the system into multiple sched domains. Then, how about providing different characteristics for each domains?

Did you actually see much improvement in any relevant workload from tweaking these parameters? If yes what did you change? And how much did it gain?

Ideally the kernel should perform well without much tweaking out of the box, simply because most users won't tweak. Adding a lot of such parameters would imply giving up on good defaults which is not a good thing.

Quoted 1 line

From what I understand they need very aggressive idle balancing; much

more so than what is normally healty.

I can see how something like that can be useful when you have a lot of very short running tasks. These could pile up on a few cpus and leave others idle.

Quoted 18 lines · Peter Zijlstra

Quoted 9 lines · Andi Kleen

Quoted 2 lines · Hidetoshi Seto

Using cpuset, now we can partition the system into multiple sched domains. Then, how about providing different characteristics for each domains?

Did you actually see much improvement in any relevant workload from tweaking these parameters? If yes what did you change? And how much did it gain?

Quoted 6 lines

Ideally the kernel should perform well without much tweaking out of the box, simply because most users won't tweak. Adding a lot of such parameters would imply giving up on good defaults which is not a good thing.

From what I understand they need very aggressive idle balancing; much

more so than what is normally healty.

Quoted 3 lines

I can see how something like that can be useful when you have a lot of very short running tasks. These could pile up on a few cpus and leave others idle.

Could the scheduler auto tune itself to this situation?

e.g. when it sees a row of very high run queue inbalances increase the frequency of the idle balancer?

Hi all,

Using cpuset, now we can partition the system into multiple sched domains. Then, how about providing different characteristics for each domains?

This patch introduces new feature of cpuset - sched domain customization.

Signed-off-by Hidetoshi Seto <[email protected]>

---
 Documentation/cpusets.txt |   89 ++++++++++++++++++++++++++++++++++++++++++++--
 1 file changed, 87 insertions(+), 2 deletions(-)

Index: GIT-torvalds/Documentation/cpusets.txt
===================================================================

Documentation/cpusets.txt +87 -3

@@ -8,6 +8,7 @@ Portions Copyright (c) 2004-2006 Silicon
 Modified by Paul Jackson <[email protected]>
 Modified by Christoph Lameter <[email protected]>
 Modified by Paul Menage <[email protected]>
+Modified by Hidetoshi Seto <[email protected]>
 
 CONTENTS:
 =========

@@ -20,7 +21,8 @@ CONTENTS:
   1.5 What is memory_pressure ?
   1.6 What is memory spread ?
   1.7 What is sched_load_balance ?
-  1.8 How do I use cpusets ?
+  1.8 What are other sched_* files ?
+  1.9 How do I use cpusets ?
 2. Usage Examples and Syntax
   2.1 Basic Usage
   2.2 Adding/removing cpus

@@ -497,7 +499,90 @@ the cpuset code to update these sched do
 partition requested with the current, and updates its sched domains,
 removing the old and adding the new, for each change.
 
-1.8 How do I use cpusets ?
+1.8 What are other sched_* files ?
+----------------------------------
+
+As described in 1.7, cpuset allows you to partition the systems CPUs
+into a number of sched domains.  Each sched domain is load balanced
+independently, in a traditional way that designed to be good for
+usual systems.
+
+But you may want to customize the behavior of load balancing for your
+special system.  For this requirement, cpuset provides some files named
+sched_* to customize the sched domain of the cpuset for some special
+situation, i.e. some specific application on some special system.
+
+These files are per-cpuset and affect the sched domain where the
+cpuset belongs to.  If multiple cpusets are overlapping and hence they
+form a single sched domain, changes in one of them affect others.
+If flag "sched_load_balance" of a cpuset is disabled, sched_* files
+have no effect since there is no sched domain belonging the cpuset.
+
+Note that modifying sched_* files will have both good and bad effects,
+and whether it is acceptable or not will be depend on your situation.
+Don't modify these files if you are not sure the effect.
+
+1.8.1 What is sched_wake_idle_far ?
+-----------------------------------
+
+When a task is woken up, scheduler try to wake up the task on idle CPU.
+
+For example, if a task A running on CPU X activates another task B
+on the same CPU X, and if CPU Y is X's sibling and performing idle,
+then scheduler migrate task B to CPU Y so that task B can start
+on CPU Y without waiting task A on CPU X.
+
+However scheduler doesn't search whole system, just searches nearby
+siblings at default.  Assume CPU Z is relatively far from CPU X.
+Even if CPU Z is idle while CPU X and the siblings are busy, scheduler
+can't migrate woken task B from X to Z.  As the result, task B on CPU X
+need to wait task A or wait load balance on the next tick.  For some
+special applications, waiting 1 tick is too long.
+
+The main reason why scheduler limits the range of searching idle CPU
+so small such as "siblings in the socket" is because it saves
+searching cost and migration cost.  Nowadays there are shared
+resources between siblings - CPU caches and so on, so this limit can
+save some migration cost assuming that the resources contain enough
+not-expired stuff for migrating task.  Usually this assumption will
+work, but not guaranteed.
+
+When the flag 'sched_wake_idle_far' is enabled, this searching range
+is expanded to all CPUs in the sched domain of the cpuset.
+
+If this flag was enabled on the example of CPU Z given above,
+scheduler can find CPU Z by taking some extra searching cost, and
+migrate task B to CPU Z by taking some extra migration cost.
+In exchange of these costs, you can start task B relatively fast.
+
+If your situation is:
+ - The migration costs between each cpu can be assumed considerably
+   small(for you) due to your special application's behavior or
+   special hardware support for CPU cache etc.
+ - The searching cost doesn't have impact(for you) or you can make
+   the searching cost enough small by managing cpuset to compact etc.
+ - The latency is required even it sacrifices cache hit rate etc.
+then turning on 'sched_wake_idle_far' would benefit you.
+
+1.8.2 What is sched_balance_newidle_far ?
+-----------------------------------------
+
+If a CPU run out of tasks in its runqueue, the CPU try to pull extra
+tasks from other busy CPUs to help them before it is going to be idle.
+
+Of course it takes some searching cost to find movable tasks,
+scheduler might not search all CPUs in the system.  For example,
+the range is limited in the same socket or node where the CPU locates.
+
+When the flag 'sched_balance_newidle_far' is enabled, this range
+is expanded to all CPUs in the sched domain of the cpuset.
+
+The assumed situation where this flag is considerable is almost same
+as that of 'sched_wake_idle_far'.  If you would like to trade better
+latency and high operating ratio in return of some other benefits,
+then enable this flag.
+
+1.9 How do I use cpusets ?
 --------------------------
 
 In order to minimize the impact of cpusets on critical kernel
--

To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Interesting ...

So, we have two flags here. One flag "sched_wake_idle_far" that will cause the current task to search farther for an idle CPU when it wakes up another task that needs a CPU on which to run, and the other flag "sched_balance_newidle_far" that will cause a soon-to-idle CPU to search farther for a task it might pull over and run, instead of going idle.

I am tempted to ask if we should not elaborate this in one dimension, and simplify it in another dimension.

First the simplification side: do we need both flags? Yes, they are two distinct cases in the code, but perhaps practical uses will always end up setting both flags the same way. If that's the case, then we are just burdening the user of these flags with understanding a detail that didn't matter to them: did a waking task or an idle CPU provoke the search? Do you have or know of a situation where you actually desire to enable one flag while disabling the other?

For the elaboration side: your proposal has just two-level's of distance, near and far. Perhaps, as architectures become more elaborate and hierarchies deeper, we would want N-level's of distance, and the ability to request such load balancing for all levels "n" for our choice of "n" <= N.

If we did both the above, then we might have a single per-cpuset file that took an integer value ... this "n". If (n == 0), that might mean no such balancing at all. If (n == 1), that might mean just the nearest balancing, for example, to the hyperthread within the same core, on some current Intel architectures. If (n == 2), then that might mean, on the same architectures, that balancing could occur across cores within the same package. If (n == 3) then that might mean, again on that architecture, that balancing could occur across packages on the same node board. As architectures evolve over time, the exact details of what each value of "n" mean would evolve, but always higher "n" would enable balancing across a wider portion of the system.

Please understand I am just brain storming here. I don't know that the alternatives I considered above are preferrable or not to what your patch presents.

Interesting ...

So, we have two flags here. One flag "sched_wake_idle_far" that will cause the current task to search farther for an idle CPU when it wakes up another task that needs a CPU on which to run, and the other flag "sched_balance_newidle_far" that will cause a soon-to-idle CPU to search farther for a task it might pull over and run, instead of going idle.

I am tempted to ask if we should not elaborate this in one dimension, and simplify it in another dimension.

First the simplification side: do we need both flags? Yes, they are two distinct cases in the code, but perhaps practical uses will always end up setting both flags the same way. If that's the case, then we are just burdening the user of these flags with understanding a detail that didn't matter to them: did a waking task or an idle CPU provoke the search? Do you have or know of a situation where you actually desire to enable one flag while disabling the other?

For the elaboration side: your proposal has just two-level's of distance, near and far. Perhaps, as architectures become more elaborate and hierarchies deeper, we would want N-level's of distance, and the ability to request such load balancing for all levels "n" for our choice of "n" <= N.

If we did both the above, then we might have a single per-cpuset file that took an integer value ... this "n". If (n == 0), that might mean no such balancing at all. If (n == 1), that might mean just the nearest balancing, for example, to the hyperthread within the same core, on some current Intel architectures. If (n == 2), then that might mean, on the same architectures, that balancing could occur across cores within the same package. If (n == 3) then that might mean, again on that architecture, that balancing could occur across packages on the same node board. As architectures evolve over time, the exact details of what each value of "n" mean would evolve, but always higher "n" would enable balancing across a wider portion of the system.

Please understand I am just brain storming here. I don't know that the alternatives I considered above are preferrable or not to what your patch presents.

Put simply, if the system tend to be idle, then "push to idle" strategy works well. OTOH if the system tend to be busy, then "pull by idle" strategy works well. Else, both strategy will work but besides of all there is a question: how much searching cost can you pay?

It looks easy... but how do you handle if cpusets are overlapping?

Quoted 4 lines · Hidetoshi

Put simply, if the system tend to be idle, then "push to idle" strategy works well. OTOH if the system tend to be busy, then "pull by idle" strategy works well. Else, both strategy will work but besides of all there is a question: how much searching cost can you pay?

So each flag has value in some cases ... that much seems reasonable to me.

But you're saying that you'd like to avoid having to turn on both, just to get the benefit of one of them, in order to avoid the searching costs of the other flag that was not valuable on that load, right?

But is this necessarily so?

If "pull by idle" is attempted on a system
which tends to be idle, then while it is true that the search for something to pull will usually find nothing, what does it matter that we wasted some otherwise idle cycles, looking for pullable, runnable tasks that cannot be found, on a system that is mostly idle?

If "push to idle" is attempted on a system that is quite busy, then couldn't that be coded to notice rather quickly if any nearby CPUs are idle, and not search if there are no idle neighbors. One could imagine a word of memory for each smaller domain ("neighborhood") of CPUs (say all the logical CPUs in a package), with one bit per logical CPU, that was set if-and-only-if that CPU was in idle. Then it would be very quick for all the CPUs in that domain to see if there are (or just were ... close enough) any idle CPUs, and skip trying to "push to idle" if that word was all zero bits. That is, there would be no sense trying to push to idle if there were no idle CPUs to push to. The only writing and the only locking of that word would be from idle loop code, and only from nearby CPUs in the same small domain, so it would not be an impediment to large system scaling or a waste of many CPU cycles on busy systems.

With a little work such as this, we could make it so that anytime you needed either flag, you could turn on both, and the other one would be harmless enough ... just a minor consumer of otherwise idle cycles.

Then with that, we could have one flag, that did both.

Quoted 1 line

It looks easy... but how do you handle if cpusets are overlapping?

Yeah - that part might be challenging. Would it work to always take the largest domain balancing requested?

I believe there are quite technical reasons why we have no "idle_map." Excellent answers would be brought by scheduler folks...

So, if there is no one can find use of two flags, I'll change it to one. Comments from any others?

I believe there are quite technical reasons why we have no "idle_map."

so I'll update my patch to take levels, getting in your suggestion.

- Add 2 new cpuset files:
sched_wake_idle_far
sched_balance_newidle_far

- Modify partition_sched_domains() and build_sched_domains() to take flags parameter passed from cpuset.

- Fill newidle_idx for node domains which currently unused but might be required for sched_balance_newidle_far.

The implementation has some updates...

Quoted 3 lines

- Add 2 new cpuset files:
sched_wake_idle_far
sched_balance_newidle_far

    -> Merged into 1 file, having levels:
         sched_relax_domain_level

Quoted 2 lines

- Modify partition_sched_domains() and build_sched_domains() to take flags parameter passed from cpuset.

    -> Changed to "attributes" rather than "flags."

Quoted 2 lines

- Fill newidle_idx for node domains which currently unused but might be required for sched_balance_newidle_far.

+ We can change the "default" level by boot option 'relax_domain_level='.

Signed-off-by Hidetoshi Seto <[email protected]>

---
 include/asm-ia64/topology.h |    2 -
 include/asm-sh/topology.h   |    2 -
 include/asm-x86/topology.h  |    2 -
 include/linux/sched.h       |   23 +++++++++++-
 kernel/cpuset.c             |   61 ++++++++++++++++++++++++++++++++
 kernel/sched.c              |   82 +++++++++++++++++++++++++++++++++++++++++---
 kernel/sched_fair.c         |    4 +-
 7 files changed, 165 insertions(+), 11 deletions(-)

Index: GIT-torvalds/include/linux/sched.h
===================================================================

include/linux/sched.h +22 -1

@@ -704,6 +704,7 @@ enum cpu_idle_type {
 #define SD_POWERSAVINGS_BALANCE	256	/* Balance for power savings */
 #define SD_SHARE_PKG_RESOURCES	512	/* Domain members share cpu pkg resources */
 #define SD_SERIALIZE		1024	/* Only a single load balancing instance */
+#define SD_WAKE_IDLE_FAR	2048	/* Gain latency sacrificing cache hit */
 
 #define BALANCE_FOR_MC_POWER	\
 	(sched_smt_power_savings ? SD_POWERSAVINGS_BALANCE : 0)

@@ -733,6 +734,24 @@ struct sched_group {
 	u32 reciprocal_cpu_power;
 };
 
+enum sched_domain_level {
+	SD_LV_NONE = 0,
+	SD_LV_SIBLING,
+	SD_LV_MC,
+	SD_LV_CPU,
+	SD_LV_NODE,
+	SD_LV_ALLNODES,
+	SD_LV_MAX
+};
+
+struct sched_domain_attr {
+	int relax_domain_level;
+};
+
+#define SD_ATTR_INIT	(struct sched_domain_attr) {	\
+	.relax_domain_level = -1,			\
+}
+
 struct sched_domain {
 	/* These fields must be setup */
 	struct sched_domain *parent;	/* top domain must be null terminated */

@@ -750,6 +769,7 @@ struct sched_domain {
 	unsigned int wake_idx;
 	unsigned int forkexec_idx;
 	int flags;			/* See SD_* */
+	enum sched_domain_level level;
 
 	/* Runtime fields. */
 	unsigned long last_balance;	/* init to jiffies. units in jiffies */

@@ -789,7 +809,8 @@ struct sched_domain {
 #endif
 };
 
-extern void partition_sched_domains(int ndoms_new, cpumask_t *doms_new);
+extern void partition_sched_domains(int ndoms_new, cpumask_t *doms_new,
+				    struct sched_domain_attr *dattr_new);
 extern int arch_reinit_sched_domains(void);
 
 #endif	/* CONFIG_SMP */

Index: GIT-torvalds/kernel/sched.c
===================================================================

kernel/sched.c +77 -5

@@ -6582,11 +6582,42 @@ static void init_sched_groups_power(int
 	} while (group != child->groups);
 }
 
+static int default_relax_domain_level = -1;
+
+static int __init setup_relax_domain_level(char *str)
+{
+	default_relax_domain_level = simple_strtoul(str, NULL, 0);
+	return 1;
+}
+__setup("relax_domain_level=", setup_relax_domain_level);
+
+static void set_domain_attribute(struct sched_domain *sd,
+				 struct sched_domain_attr *attr)
+{
+	int request;
+
+	if (!attr || attr->relax_domain_level < 0) {
+		if (default_relax_domain_level < 0)
+			return;
+		else
+			request = default_relax_domain_level;
+	} else
+		request = attr->relax_domain_level;
+	if (request < sd->level) {
+		/* turn off idle balance on this domain */
+		sd->flags &= ~(SD_WAKE_IDLE|SD_BALANCE_NEWIDLE);
+	} else {
+		/* turn on idle balance on this domain */
+		sd->flags |= (SD_WAKE_IDLE_FAR|SD_BALANCE_NEWIDLE);
+	}
+}
+
 /*
  * Build sched domains for a given set of cpus and attach the sched domains
  * to the individual cpus
  */
-static int build_sched_domains(const cpumask_t *cpu_map)
+static int __build_sched_domains(const cpumask_t *cpu_map,
+				 struct sched_domain_attr *attr)
 {
 	int i;
 	struct root_domain *rd;

@@ -6626,7 +6657,9 @@ static int build_sched_domains(const cpu
 				SD_NODES_PER_DOMAIN*cpus_weight(nodemask)) {
 			sd = &per_cpu(allnodes_domains, i);
 			*sd = SD_ALLNODES_INIT;
+			sd->level = SD_LV_ALLNODES;
 			sd->span = *cpu_map;
+			set_domain_attribute(sd, attr);
 			cpu_to_allnodes_group(i, cpu_map, &sd->groups);
 			p = sd;
 			sd_allnodes = 1;

@@ -6635,7 +6668,9 @@ static int build_sched_domains(const cpu
 
 		sd = &per_cpu(node_domains, i);
 		*sd = SD_NODE_INIT;
+		sd->level = SD_LV_NODE;
 		sd->span = sched_domain_node_span(cpu_to_node(i));
+		set_domain_attribute(sd, attr);
 		sd->parent = p;
 		if (p)
 			p->child = sd;

@@ -6645,7 +6680,9 @@ static int build_sched_domains(const cpu
 		p = sd;
 		sd = &per_cpu(phys_domains, i);
 		*sd = SD_CPU_INIT;
+		sd->level = SD_LV_CPU;
 		sd->span = nodemask;
+		set_domain_attribute(sd, attr);
 		sd->parent = p;
 		if (p)
 			p->child = sd;

@@ -6655,8 +6692,10 @@ static int build_sched_domains(const cpu
 		p = sd;
 		sd = &per_cpu(core_domains, i);
 		*sd = SD_MC_INIT;
+		sd->level = SD_LV_MC;
 		sd->span = cpu_coregroup_map(i);
 		cpus_and(sd->span, sd->span, *cpu_map);
+		set_domain_attribute(sd, attr);
 		sd->parent = p;
 		p->child = sd;
 		cpu_to_core_group(i, cpu_map, &sd->groups);

@@ -6666,8 +6705,10 @@ static int build_sched_domains(const cpu
 		p = sd;
 		sd = &per_cpu(cpu_domains, i);
 		*sd = SD_SIBLING_INIT;
+		sd->level = SD_LV_SIBLING;
 		sd->span = per_cpu(cpu_sibling_map, i);
 		cpus_and(sd->span, sd->span, *cpu_map);
+		set_domain_attribute(sd, attr);
 		sd->parent = p;
 		p->child = sd;
 		cpu_to_cpu_group(i, cpu_map, &sd->groups);

@@ -6840,8 +6881,15 @@ error:
 #endif
 }
 
+static int build_sched_domains(const cpumask_t *cpu_map)
+{
+	return __build_sched_domains(cpu_map, NULL);
+}
+
 static cpumask_t *doms_cur;	/* current sched domains */
 static int ndoms_cur;		/* number of sched domains in 'doms_cur' */
+static struct sched_domain_attr *dattr_cur;	/* attribues of custom domains
+						   in 'doms_cur' */
 
 /*
  * Special case: If a kmalloc of a doms_cur partition (array of

@@ -6868,6 +6916,7 @@ static int arch_init_sched_domains(const
 	doms_cur = kmalloc(sizeof(cpumask_t), GFP_KERNEL);
 	if (!doms_cur)
 		doms_cur = &fallback_doms;
+	dattr_cur = NULL;
 	cpus_andnot(*doms_cur, *cpu_map, cpu_isolated_map);
 	err = build_sched_domains(doms_cur);
 	register_sched_domain_sysctl();

@@ -6896,6 +6945,22 @@ static void detach_destroy_domains(const
 	arch_destroy_sched_domains(cpu_map);
 }
 
+/* handle null as "default" */
+static int dattrs_equal(struct sched_domain_attr *cur, int idx_cur,
+			struct sched_domain_attr *new, int idx_new)
+{
+	struct sched_domain_attr tmp;
+
+	/* fast path */
+	if (!new && !cur)
+		return 1;
+
+	tmp = SD_ATTR_INIT;
+	return !memcmp(cur ? (cur + idx_cur) : &tmp,
+			new ? (new + idx_new) : &tmp,
+			sizeof(struct sched_domain_attr));
+}
+
 /*
  * Partition sched domains as specified by the 'ndoms_new'
  * cpumasks in the array doms_new[] of cpumasks. This compares

@@ -6917,7 +6982,8 @@ static void detach_destroy_domains(const
  *
  * Call with hotplug lock held
  */
-void partition_sched_domains(int ndoms_new, cpumask_t *doms_new)
+void partition_sched_domains(int ndoms_new, cpumask_t *doms_new,
+			     struct sched_domain_attr *dattr_new)
 {
 	int i, j;
 

@@ -6929,13 +6995,15 @@ void partition_sched_domains(int ndoms_n
 	if (doms_new == NULL) {
 		ndoms_new = 1;
 		doms_new = &fallback_doms;
+		dattr_new = NULL;
 		cpus_andnot(doms_new[0], cpu_online_map, cpu_isolated_map);
 	}
 
 	/* Destroy deleted domains */
 	for (i = 0; i < ndoms_cur; i++) {
 		for (j = 0; j < ndoms_new; j++) {
-			if (cpus_equal(doms_cur[i], doms_new[j]))
+			if (cpus_equal(doms_cur[i], doms_new[j])
+			    && dattrs_equal(dattr_cur, i, dattr_new, j))
 				goto match1;
 		}
 		/* no match - a current sched domain not in new doms_new[] */

@@ -6947,11 +7015,13 @@ match1:
 	/* Build new domains */
 	for (i = 0; i < ndoms_new; i++) {
 		for (j = 0; j < ndoms_cur; j++) {
-			if (cpus_equal(doms_new[i], doms_cur[j]))
+			if (cpus_equal(doms_new[i], doms_cur[j])
+			    && dattrs_equal(dattr_new, i, dattr_cur, j))
 				goto match2;
 		}
 		/* no match - add a new doms_new */
-		build_sched_domains(doms_new + i);
+		__build_sched_domains(doms_new + i,
+					dattr_new ? dattr_new + i : NULL);
 match2:
 		;
 	}

@@ -6959,7 +7029,9 @@ match2:
 	/* Remember the new sched domains */
 	if (doms_cur != &fallback_doms)
 		kfree(doms_cur);
+	kfree(dattr_cur);	/* kfree(NULL) is safe */
 	doms_cur = doms_new;
+	dattr_cur = dattr_new;
 	ndoms_cur = ndoms_new;
 
 	register_sched_domain_sysctl();

Index: GIT-torvalds/kernel/sched_fair.c
===================================================================

kernel/sched_fair.c +3 -1

@@ -957,7 +957,9 @@ static int wake_idle(int cpu, struct tas
 		return cpu;
 
 	for_each_domain(cpu, sd) {
-		if (sd->flags & SD_WAKE_IDLE) {
+		if ((sd->flags & SD_WAKE_IDLE)
+		    || ((sd->flags & SD_WAKE_IDLE_FAR)
+			&& !task_hot(p, task_rq(p)->clock, sd))) {
 			cpus_and(tmp, sd->span, p->cpus_allowed);
 			for_each_cpu_mask(i, tmp) {
 				if (idle_cpu(i)) {

Index: GIT-torvalds/kernel/cpuset.c
===================================================================

kernel/cpuset.c +60 -1

@@ -98,6 +98,9 @@ struct cpuset {
 	/* partition number for rebuild_sched_domains() */
 	int pn;
 
+	/* for custom sched domain */
+	int relax_domain_level;
+
 	/* used for walking a cpuset heirarchy */
 	struct list_head stack_list;
 };

@@ -478,6 +481,16 @@ static int cpusets_overlap(struct cpuset
 	return cpus_intersects(a->cpus_allowed, b->cpus_allowed);
 }
 
+static void
+update_domain_attr(struct sched_domain_attr *dattr, struct cpuset *c)
+{
+	if (!dattr)
+		return;
+	if (dattr->relax_domain_level < c->relax_domain_level)
+		dattr->relax_domain_level = c->relax_domain_level;
+	return;
+}
+
 /*
  * rebuild_sched_domains()
  *

@@ -553,12 +566,14 @@ static void rebuild_sched_domains(void)
 	int csn;		/* how many cpuset ptrs in csa so far */
 	int i, j, k;		/* indices for partition finding loops */
 	cpumask_t *doms;	/* resulting partition; i.e. sched domains */
+	struct sched_domain_attr *dattr;  /* attributes for custom domains */
 	int ndoms;		/* number of sched domains in result */
 	int nslot;		/* next empty doms[] cpumask_t slot */
 
 	q = NULL;
 	csa = NULL;
 	doms = NULL;
+	dattr = NULL;
 
 	/* Special case for the 99% of systems with one, full, sched domain */
 	if (is_sched_load_balance(&top_cpuset)) {

@@ -566,6 +581,11 @@ static void rebuild_sched_domains(void)
 		doms = kmalloc(sizeof(cpumask_t), GFP_KERNEL);
 		if (!doms)
 			goto rebuild;
+		dattr = kmalloc(sizeof(struct sched_domain_attr), GFP_KERNEL);
+		if (dattr) {
+			*dattr = SD_ATTR_INIT;
+			update_domain_attr(dattr, &top_cpuset);
+		}
 		*doms = top_cpuset.cpus_allowed;
 		goto rebuild;
 	}

@@ -622,6 +642,7 @@ restart:
 	doms = kmalloc(ndoms * sizeof(cpumask_t), GFP_KERNEL);
 	if (!doms)
 		goto rebuild;
+	dattr = kmalloc(ndoms * sizeof(struct sched_domain_attr), GFP_KERNEL);
 
 	for (nslot = 0, i = 0; i < csn; i++) {
 		struct cpuset *a = csa[i];

@@ -644,12 +665,15 @@ restart:
 			}
 
 			cpus_clear(*dp);
+			if (dattr)
+				*(dattr + nslot) = SD_ATTR_INIT;
 			for (j = i; j < csn; j++) {
 				struct cpuset *b = csa[j];
 
 				if (apn == b->pn) {
 					cpus_or(*dp, *dp, b->cpus_allowed);
 					b->pn = -1;
+					update_domain_attr(dattr, b);
 				}
 			}
 			nslot++;

@@ -660,7 +684,7 @@ restart:
 rebuild:
 	/* Have scheduler rebuild sched domains */
 	get_online_cpus();
-	partition_sched_domains(ndoms, doms);
+	partition_sched_domains(ndoms, doms, dattr);
 	put_online_cpus();
 
 done:

@@ -668,6 +692,7 @@ done:
 		kfifo_free(q);
 	kfree(csa);
 	/* Don't kfree(doms) -- partition_sched_domains() does that. */
+	/* Don't kfree(dattr) -- partition_sched_domains() does that. */
 }
 
 static inline int started_after_time(struct task_struct *t1,

@@ -1011,6 +1036,21 @@ static int update_memory_pressure_enable
 	return 0;
 }
 
+static int update_relax_domain_level(struct cpuset *cs, char *buf)
+{
+	int val = simple_strtol(buf, NULL, 10);
+
+	if (val < 0)
+		val = -1;
+
+	if (val != cs->relax_domain_level) {
+		cs->relax_domain_level = val;
+		rebuild_sched_domains();
+	}
+
+	return 0;
+}
+
 /*
  * update_flag - read a 0 or a 1 in a file and update associated flag
  * bit:	the bit to update (CS_CPU_EXCLUSIVE, CS_MEM_EXCLUSIVE,

@@ -1202,6 +1242,7 @@ typedef enum {
 	FILE_CPU_EXCLUSIVE,
 	FILE_MEM_EXCLUSIVE,
 	FILE_SCHED_LOAD_BALANCE,
+	FILE_SCHED_RELAX_DOMAIN_LEVEL,
 	FILE_MEMORY_PRESSURE_ENABLED,
 	FILE_MEMORY_PRESSURE,
 	FILE_SPREAD_PAGE,

@@ -1256,6 +1297,9 @@ static ssize_t cpuset_common_file_write(
 	case FILE_SCHED_LOAD_BALANCE:
 		retval = update_flag(CS_SCHED_LOAD_BALANCE, cs, buffer);
 		break;
+	case FILE_SCHED_RELAX_DOMAIN_LEVEL:
+		retval = update_relax_domain_level(cs, buffer);
+		break;
 	case FILE_MEMORY_MIGRATE:
 		retval = update_flag(CS_MEMORY_MIGRATE, cs, buffer);
 		break;

@@ -1354,6 +1398,9 @@ static ssize_t cpuset_common_file_read(s
 	case FILE_SCHED_LOAD_BALANCE:
 		*s++ = is_sched_load_balance(cs) ? '1' : '0';
 		break;
+	case FILE_SCHED_RELAX_DOMAIN_LEVEL:
+		s += sprintf(s, "%d", cs->relax_domain_level);
+		break;
 	case FILE_MEMORY_MIGRATE:
 		*s++ = is_memory_migrate(cs) ? '1' : '0';
 		break;

@@ -1424,6 +1471,13 @@ static struct cftype cft_sched_load_bala
 	.private = FILE_SCHED_LOAD_BALANCE,
 };
 
+static struct cftype cft_sched_relax_domain_level = {
+	.name = "sched_relax_domain_level",
+	.read = cpuset_common_file_read,
+	.write = cpuset_common_file_write,
+	.private = FILE_SCHED_RELAX_DOMAIN_LEVEL,
+};
+
 static struct cftype cft_memory_migrate = {
 	.name = "memory_migrate",
 	.read = cpuset_common_file_read,

@@ -1475,6 +1529,9 @@ static int cpuset_populate(struct cgroup
 		return err;
 	if ((err = cgroup_add_file(cont, ss, &cft_sched_load_balance)) < 0)
 		return err;
+	if ((err = cgroup_add_file(cont, ss,
+					&cft_sched_relax_domain_level)) < 0)
+		return err;
 	if ((err = cgroup_add_file(cont, ss, &cft_memory_pressure)) < 0)
 		return err;
 	if ((err = cgroup_add_file(cont, ss, &cft_spread_page)) < 0)

@@ -1559,6 +1616,7 @@ static struct cgroup_subsys_state *cpuse
 	cs->mems_allowed = NODE_MASK_NONE;
 	cs->mems_generation = cpuset_mems_generation++;
 	fmeter_init(&cs->fmeter);
+	cs->relax_domain_level = -1;
 
 	cs->parent = parent;
 	number_of_cpusets++;

@@ -1631,6 +1689,7 @@ int __init cpuset_init(void)
 	fmeter_init(&top_cpuset.fmeter);
 	top_cpuset.mems_generation = cpuset_mems_generation++;
 	set_bit(CS_SCHED_LOAD_BALANCE, &top_cpuset.flags);
+	top_cpuset.relax_domain_level = -1;
 
 	err = register_filesystem(&cpuset_fs_type);
 	if (err < 0)

Index: GIT-torvalds/include/asm-ia64/topology.h
===================================================================

include/asm-ia64/topology.h +1 -1

@@ -93,7 +93,7 @@ void build_cpu_to_node_map(void);
 	.cache_nice_tries	= 2,			\
 	.busy_idx		= 3,			\
 	.idle_idx		= 2,			\
-	.newidle_idx		= 0, /* unused */	\
+	.newidle_idx		= 2,			\
 	.wake_idx		= 1,			\
 	.forkexec_idx		= 1,			\
 	.flags			= SD_LOAD_BALANCE	\

Index: GIT-torvalds/include/asm-sh/topology.h
===================================================================

include/asm-sh/topology.h +1 -1

@@ -16,7 +16,7 @@
 	.cache_nice_tries	= 2,			\
 	.busy_idx		= 3,			\
 	.idle_idx		= 2,			\
-	.newidle_idx		= 0,			\
+	.newidle_idx		= 2,			\
 	.wake_idx		= 1,			\
 	.forkexec_idx		= 1,			\
 	.flags			= SD_LOAD_BALANCE	\

Index: GIT-torvalds/include/asm-x86/topology.h
===================================================================

include/asm-x86/topology.h +1 -1

@@ -129,7 +129,7 @@ extern unsigned long node_remap_size[];
 
 # define SD_CACHE_NICE_TRIES	2
 # define SD_IDLE_IDX		2
-# define SD_NEWIDLE_IDX		0
+# define SD_NEWIDLE_IDX		2
 # define SD_FORKEXEC_IDX	1
 
 #endif

This seems like a sufficiently flexible interface. Paul, have you got any outstanding objections?

Acked-by Peter Zijlstra <[email protected]>

Paul, could you give me your response?