2.6.3-rc3-mm1: sched-group-power

The following patch implements a cpu_power member to struct sched_group.

This allows special casing to be removed for SMT groups in the balancing code. It does not take CPU hotplug into account yet, but that shouldn't be too hard.

I have tested it on the NUMAQ by pretending it has SMT. Works as expected. Active balances across nodes.

Andrew, please apply. I suppose you'd better not send the scheduler changes to Linus quite yet.

 linux-2.6-npiggin/arch/i386/kernel/smpboot.c |   15 ++-
 linux-2.6-npiggin/include/linux/sched.h      |   12 ++
 linux-2.6-npiggin/kernel/sched.c             |  131 ++++++++++++++-------------
 3 files changed, 91 insertions(+), 67 deletions(-)

diff -puN include/linux/sched.h~sched-group-power include/linux/sched.h
@@ -530,15 +530,25 @@ do { if (atomic_dec_and_test(&(tsk)->usa
 #define PF_SYNCWRITE	0x00200000	/* I am doing a sync write */
 
 #ifdef CONFIG_SMP
+#define SCHED_LOAD_SHIFT 7	/* increase resolution of load calculations */
+#define SCHED_LOAD_SCALE (1 << SCHED_LOAD_SHIFT)
+
 #define SD_FLAG_NEWIDLE		1	/* Balance when about to become idle */
 #define SD_FLAG_EXEC		2	/* Balance on exec */
 #define SD_FLAG_WAKE		4	/* Balance on task wakeup */
 #define SD_FLAG_FASTMIGRATE	8	/* Sync wakes put task on waking CPU */
-#define SD_FLAG_IDLE		16	/* Should not have all CPUs idle */
 
 struct sched_group {
 	struct sched_group *next;	/* Must be a circular list */
 	cpumask_t cpumask;
+
+	/*
+	 * CPU power of this group, SCHED_LOAD_SCALE being max power for a
+	 * single CPU. This should be read only (except for setup). Although
+	 * it will need to be written to at cpu hot(un)plug time, perhaps the
+	 * cpucontrol semaphore will provide enough exclusion?
+	 */
+	unsigned long cpu_power;
 };
 
 struct sched_domain {

diff -puN arch/i386/kernel/smpboot.c~sched-group-power arch/i386/kernel/smpboot.c
@@ -1149,7 +1149,6 @@ __init void arch_init_sched_domains(void
 
 		*phys_domain = SD_CPU_INIT;
 		phys_domain->span = nodemask;
-		phys_domain->flags |= SD_FLAG_IDLE;
 
 		*node_domain = SD_NODE_INIT;
 		node_domain->span = cpu_online_map;

@@ -1169,6 +1168,7 @@ __init void arch_init_sched_domains(void
 
 			cpu->cpumask = CPU_MASK_NONE;
 			cpu_set(j, cpu->cpumask);
+			cpu->cpu_power = SCHED_LOAD_SCALE;
 
 			if (!first_cpu)
 				first_cpu = cpu;

@@ -1182,6 +1182,7 @@ __init void arch_init_sched_domains(void
 	for (i = 0; i < MAX_NUMNODES; i++) {
 		int j;
 		cpumask_t nodemask;
+		struct sched_group *node = &sched_group_nodes[i];
 		cpus_and(nodemask, node_to_cpumask(i), cpu_online_map);
 
 		if (cpus_empty(nodemask))

@@ -1197,6 +1198,12 @@ __init void arch_init_sched_domains(void
 				continue;
 
 			cpu->cpumask = cpu_domain->span;
+			/*
+			 * Make each extra sibling increase power by 10% of
+			 * the basic CPU. This is very arbitrary.
+			 */
+			cpu->cpu_power = SCHED_LOAD_SCALE + SCHED_LOAD_SCALE*(cpus_weight(cpu->cpumask)-1) / 10;
+			node->cpu_power += cpu->cpu_power;
 
 			if (!first_cpu)
 				first_cpu = cpu;

@@ -1218,6 +1225,7 @@ __init void arch_init_sched_domains(void
 			continue;
 
 		cpu->cpumask = nodemask;
+		/* ->cpu_power already setup */
 
 		if (!first_cpu)
 			first_cpu = cpu;

@@ -1227,7 +1235,6 @@ __init void arch_init_sched_domains(void
 	}
 	last_cpu->next = first_cpu;
 
-
 	mb();
 	for_each_cpu_mask(i, cpu_online_map) {
 		int node = cpu_to_node(i);

@@ -1265,7 +1272,6 @@ __init void arch_init_sched_domains(void
 
 		*phys_domain = SD_CPU_INIT;
 		phys_domain->span = cpu_online_map;
-		phys_domain->flags |= SD_FLAG_IDLE;
 	}
 
 	/* Set up CPU (sibling) groups */

@@ -1282,6 +1288,7 @@ __init void arch_init_sched_domains(void
 
 			cpus_clear(cpu->cpumask);
 			cpu_set(j, cpu->cpumask);
+			cpu->cpu_power = SCHED_LOAD_SCALE;
 
 			if (!first_cpu)
 				first_cpu = cpu;

@@ -1302,6 +1309,8 @@ __init void arch_init_sched_domains(void
 			continue;
 
 		cpu->cpumask = cpu_domain->span;
+		/* See SMT+NUMA setup for comment */
+		cpu->cpu_power = SCHED_LOAD_SCALE + SCHED_LOAD_SCALE*(cpus_weight(cpu->cpumask)-1) / 10;
 
 		if (!first_cpu)
 			first_cpu = cpu;

_

Nick, I'm not sure what capability this patch adds .. perhaps some words of explanation.

So we have SMT/HT situations where we'd prefer to balance across cores; that is, if 0, 1, 2, and 3 share a core and 4, 5, 6, and 7 share a core, you'd like two processes to arrange themselves so one is on [0123] and another is on [4567]. This is what the SD_IDLE flag indicated before.

With this patch, we can "weight" the load imposed by certain cpus, right? What advantage does this give us? On a given machine, won't the "weight" of any one set of SMT siblings and cores be uniform with respect to all the cores and siblings anyway?

Rick

It is difficult to propogate the SD_FLAG_IDLE attribute up multiple domains.

For example, with SMT + CPU + NODE domains you can get into the following situation:

01, 23 are 4 siblings in 2 cores on node 0,
45, 67 are " " " on node 1.

The top level balancing domain now spans 01234567, and wants to balance between groups 0123, and 4567. We don't want SD_FLAG_IDLE semantics here, because that would mean if two tasks were running on node 0, one would be migrated to node 1. We want to migrate 1 task if one node is idle, and the other has 3 processes running for example.

Also this copes with siblings becoming much more powerful, or some groups with SMT turned off, some on (think hotplug cpu), different speed CPUs, etc.

So let me try a diagram. Each of these groups of numbers represent a cpu_group, and the labels to the left are individual sched_domains.

SD1 01234567
SD2-SD3 0123 4567
SD4-SD7 01 23 45 67
SD8-SD15 0 1 2 3 4 5 6 7

Currently, we assume each cpu has a power of 1, so each cpu group in domains SD8-SD15 would have a power of 1, each cpu group in SD4-SD7 would have a power of 2, each of SD2 and SD3 would have a power of 4, and collectively, all CPUs as represented in SD1 would have a power of 8. Of course, we don't really make use of this assumption but this just enumerates our assumption that all nodes, all cpus are created equal.

Your new power code would assign each cpu group a static power other than this, making SMT pairs, for instance, 1.2 instead of 2. In the case of four siblings, 1.4 instead of 4. Correct? In the example above, SD2 and SD3 would have a power rating of 2.4, and SD1 would have a power rating of 4*1.2 or 4.8, right?

With your current code, we only consult the power ratings if we've already decided that we are currently "balanced enough". I'd go one step further and say that manipulating for power only makes sense if you have an idle processor somewhere. If all processors are busy, then short of some quality-of-process assessment, how can you improve power? (You could improve fairness, I suppose, but that would require lots more stats and history than we have here.) If one set of procs is slower than another, won't that make itself apparent by a longer queue developing there? (or shorter queues forming somewhere else?) and it being load-balanced by the existing algorithm? Seems to me we only need to make power decisions when we want to consider an idle processor stealing a task (a possibly *running* task) from another processor because this processor is faster/stronger/better.

Rick

Well we used to sum up the number of CPUs in each group, so it wasn't quite that bad. We assumed all CPUs are created equal.

Right.

Well we do work out the per group loads by dividing with the power rating instead of cpus-in-the-group too.

Yeah, probably we could change that test to:
if (*imbalance <= SCHED_LOAD_SCALE / 2

           && this_load < SCHED_LOAD_SCALE)

Either way, if the calculation should be done in such a way that if your CPUs are not idle, then it wouldn't predict a performance increase.

No doubt there is room for improvement, but hopefully it is now at a "good enough" stage...