Skip to contentSkip to conversation

top level scheduler domain for ia64

PATCH In review Jesse Barnes 9 participants 27 messages Oct 19, 2004 – Nov 8, 2004

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 top level scheduler domain for ia64

Some have noticed that the overlapping sched domains code doesn't quite work as intended (it results in disjoint domains on some machines), and that a top level, machine spanning domain is needed. This patch from John Hawkes adds it to the ia64 code. This allows processes to run on all CPUs in large systems, though balancing is limited. It should go to Linus soon now otherwise large systems will only have ~16p (depending on topology) usable by the scheduler. I sanity checked it on a small system after rediffing John's original, and he's done some testing on very large systems.

Nick, can you buy off on the sched.c change? Alternatively, do you want to send that fix separately John?

Thanks,
Jesse

===== arch/ia64/kernel/domain.c 1.3 vs edited =====

arch/ia64/kernel/domain.c +32 -0
@@ -119,6 +119,14 @@
  */
 static DEFINE_PER_CPU(struct sched_domain, node_domains);
 static struct sched_group *sched_group_nodes[MAX_NUMNODES];
+
+static DEFINE_PER_CPU(struct sched_domain, allnodes_domains);
+static struct sched_group sched_group_allnodes[MAX_NUMNODES];
+
+static int __devinit cpu_to_allnodes_group(int cpu)
+{
+	return cpu_to_node(cpu);
+}
 #endif
 
 /*
@@ -149,9 +157,21 @@
 		cpus_and(nodemask, nodemask, cpu_default_map);
 
 #ifdef CONFIG_NUMA
+		if (num_online_cpus()
+				> SD_NODES_PER_DOMAIN*cpus_weight(nodemask)) {
+			sd = &per_cpu(allnodes_domains, i);
+			*sd = SD_ALLNODES_INIT;
+			sd->span = cpu_default_map;
+			group = cpu_to_allnodes_group(i);
+			sd->groups = &sched_group_allnodes[group];
+			p = sd;
+		} else
+			p = NULL;
+
 		sd = &per_cpu(node_domains, i);
 		*sd = SD_NODE_INIT;
 		sd->span = sched_domain_node_span(node);
+		sd->parent = p;
 		cpus_and(sd->span, sd->span, cpu_default_map);
 #endif
 
@@ -201,6 +221,9 @@
 	}
 
 #ifdef CONFIG_NUMA
+	init_sched_build_groups(sched_group_allnodes, cpu_default_map,
+				&cpu_to_allnodes_group);
+
 	for (i = 0; i < MAX_NUMNODES; i++) {
 		/* Set up node groups */
 		struct sched_group *sg, *prev;
@@ -282,6 +305,15 @@
 		power = SCHED_LOAD_SCALE + SCHED_LOAD_SCALE *
 				(cpus_weight(sd->groups->cpumask)-1) / 10;
 		sd->groups->cpu_power = power;
+
+#ifdef CONFIG_NUMA
+		sd = &per_cpu(allnodes_domains, i);
+		if (sd->groups) {
+			power = SCHED_LOAD_SCALE + SCHED_LOAD_SCALE *
+				(cpus_weight(sd->groups->cpumask)-1) / 10;
+			sd->groups->cpu_power = power;
+		}
+#endif
 	}
 
 #ifdef CONFIG_NUMA

===== include/asm-ia64/topology.h 1.12 vs edited =====

include/asm-ia64/topology.h +20 -1
@@ -58,7 +58,26 @@
 				| SD_BALANCE_EXEC	\
 				| SD_WAKE_BALANCE,	\
 	.last_balance		= jiffies,		\
-	.balance_interval	= 10,			\
+	.balance_interval	= 1,			\
+	.nr_balance_failed	= 0,			\
+}
+
+/* sched_domains SD_ALLNODES_INIT for IA64 NUMA machines */
+#define SD_ALLNODES_INIT (struct sched_domain) {	\
+	.span			= CPU_MASK_NONE,	\
+	.parent			= NULL,			\
+	.groups			= NULL,			\
+	.min_interval		= 80,			\
+	.max_interval		= 320,			\
+	.busy_factor		= 320,			\
+	.imbalance_pct		= 125,			\
+	.cache_hot_time		= (10*1000000),		\
+	.cache_nice_tries	= 1,			\
+	.per_cpu_gain		= 100,			\
+	.flags			= SD_LOAD_BALANCE	\
+				| SD_BALANCE_EXEC,	\
+	.last_balance		= jiffies,		\
+	.balance_interval	= 100*(63+num_online_cpus())/64,   \
 	.nr_balance_failed	= 0,			\
 }
 

===== kernel/sched.c 1.367 vs edited =====

kernel/sched.c +1 -2
@@ -4378,11 +4378,10 @@
 			printk("domain %d: ", level);
 
 			if (!(sd->flags & SD_LOAD_BALANCE)) {
-				printk("does not balance");
+				printk("does not load-balance");
 				if (sd->parent)
 					printk(" ERROR !SD_LOAD_BALANCE domain has parent");
 				printk("\n");
-				break;
 			}
 
 			printk("span %s\n", str);
#1 lore raw
Quoted 11 lines · Jesse Barnes

Some have noticed that the overlapping sched domains code doesn't quite work as intended (it results in disjoint domains on some machines), and that a top level, machine spanning domain is needed. This patch from John Hawkes adds it to the ia64 code. This allows processes to run on all CPUs in large systems, though balancing is limited. It should go to Linus soon now otherwise large systems will only have ~16p (depending on topology) usable by the scheduler. I sanity checked it on a small system after rediffing John's original, and he's done some testing on very large systems.

Nick, can you buy off on the sched.c change? Alternatively, do you want to send that fix separately John?

No, that looks good. It is a pretty trivial change so I think its harmless to go in with this patch.

I'd consider increasing balancing rates for the SD_NODE_INIT domains after this patch goes in too.

#2 lore raw
In reply to Jesse Barnes
Quoted 4 lines

Some have noticed that the overlapping sched domains code doesn't quite work as intended (it results in disjoint domains on some machines), and that a top level, machine spanning domain is needed.

Why is the solution to jam this into the ia64 specific code? From this description it sounds like a generic scheduler problem, so the solution ought to be up in some generic code.

+ .min_interval = 80, \
+ .max_interval = 320, \
+ .busy_factor = 320, \
+ .imbalance_pct = 125, \
+ .cache_hot_time = (10*1000000), \
+ .balance_interval = 100*(63+num_online_cpus())/64, \

That's a lot of magic numbers and formulae ... are they right? How would a user know if they are right.

Quoted 2 lines

Nick, can you buy off on the sched.c change? Alternatively, do you want to send that fix separately John?

I saw the ACK from Nick ... but kernel/sched.c changes will have to go through Andrew, not me. Are the ia64 and generic parts separable? If the sched.c change goes in, do other architectures need to have some equivalent change?

-Tony

===== arch/ia64/kernel/domain.c 1.3 vs edited =====

arch/ia64/kernel/domain.c +32 -0
@@ -119,6 +119,14 @@
  */
 static DEFINE_PER_CPU(struct sched_domain, node_domains);
 static struct sched_group *sched_group_nodes[MAX_NUMNODES];
+
+static DEFINE_PER_CPU(struct sched_domain, allnodes_domains);
+static struct sched_group sched_group_allnodes[MAX_NUMNODES];
+
+static int __devinit cpu_to_allnodes_group(int cpu)
+{
+	return cpu_to_node(cpu);
+}
 #endif
 
 /*
@@ -149,9 +157,21 @@
 		cpus_and(nodemask, nodemask, cpu_default_map);
 
 #ifdef CONFIG_NUMA
+		if (num_online_cpus()
+				> SD_NODES_PER_DOMAIN*cpus_weight(nodemask)) {
+			sd = &per_cpu(allnodes_domains, i);
+			*sd = SD_ALLNODES_INIT;
+			sd->span = cpu_default_map;
+			group = cpu_to_allnodes_group(i);
+			sd->groups = &sched_group_allnodes[group];
+			p = sd;
+		} else
+			p = NULL;
+
 		sd = &per_cpu(node_domains, i);
 		*sd = SD_NODE_INIT;
 		sd->span = sched_domain_node_span(node);
+		sd->parent = p;
 		cpus_and(sd->span, sd->span, cpu_default_map);
 #endif
 
@@ -201,6 +221,9 @@
 	}
 
 #ifdef CONFIG_NUMA
+	init_sched_build_groups(sched_group_allnodes, cpu_default_map,
+				&cpu_to_allnodes_group);
+
 	for (i = 0; i < MAX_NUMNODES; i++) {
 		/* Set up node groups */
 		struct sched_group *sg, *prev;
@@ -282,6 +305,15 @@
 		power = SCHED_LOAD_SCALE + SCHED_LOAD_SCALE *
 				(cpus_weight(sd->groups->cpumask)-1) / 10;
 		sd->groups->cpu_power = power;
+
+#ifdef CONFIG_NUMA
+		sd = &per_cpu(allnodes_domains, i);
+		if (sd->groups) {
+			power = SCHED_LOAD_SCALE + SCHED_LOAD_SCALE *
+				(cpus_weight(sd->groups->cpumask)-1) / 10;
+			sd->groups->cpu_power = power;
+		}
+#endif
 	}
 
 #ifdef CONFIG_NUMA

===== include/asm-ia64/topology.h 1.12 vs edited =====

include/asm-ia64/topology.h +20 -1
@@ -58,7 +58,26 @@
 				| SD_BALANCE_EXEC	\
 				| SD_WAKE_BALANCE,	\
 	.last_balance		= jiffies,		\
-	.balance_interval	= 10,			\
+	.balance_interval	= 1,			\
+	.nr_balance_failed	= 0,			\
+}
+
+/* sched_domains SD_ALLNODES_INIT for IA64 NUMA machines */
+#define SD_ALLNODES_INIT (struct sched_domain) {	\
+	.span			= CPU_MASK_NONE,	\
+	.parent			= NULL,			\
+	.groups			= NULL,			\
+	.min_interval		= 80,			\
+	.max_interval		= 320,			\
+	.busy_factor		= 320,			\
+	.imbalance_pct		= 125,			\
+	.cache_hot_time		= (10*1000000),		\
+	.cache_nice_tries	= 1,			\
+	.per_cpu_gain		= 100,			\
+	.flags			= SD_LOAD_BALANCE	\
+				| SD_BALANCE_EXEC,	\
+	.last_balance		= jiffies,		\
+	.balance_interval	= 100*(63+num_online_cpus())/64,   \
 	.nr_balance_failed	= 0,			\
 }
 

===== kernel/sched.c 1.367 vs edited =====

kernel/sched.c +1 -2
@@ -4378,11 +4378,10 @@
 			printk("domain %d: ", level);
 
 			if (!(sd->flags & SD_LOAD_BALANCE)) {
-				printk("does not balance");
+				printk("does not load-balance");
 				if (sd->parent)
 					printk(" ERROR !SD_LOAD_BALANCE domain has parent");
 				printk("\n");
-				break;
 			}
 
 			printk("span %s\n", str);
#3 lore raw
In reply to Jesse Barnes
Quoted 10 lines · Tony
Quoted 4 lines

Some have noticed that the overlapping sched domains code doesn't quite work as intended (it results in disjoint domains on some machines), and that a top level, machine spanning domain is needed.

Why is the solution to jam this into the ia64 specific code? From this description it sounds like a generic scheduler problem, so the solution ought to be up in some generic code.

So far, ia64 is the only one that uses these overlapping domains. The code to build them is complex enough that we didn't want to put it into the generic code at the moment (but it is fairly well contained, all just in arch/ia64/kernel/domain.c and asm/topology.h).

Quoted 10 lines
include/asm-ia64/topology.h +6 -0
+	.min_interval		= 80,			\
+	.max_interval		= 320,			\
+	.busy_factor		= 320,			\
+	.imbalance_pct		= 125,			\
+	.cache_hot_time		= (10*1000000),		\
+	.balance_interval	= 100*(63+num_online_cpus())/64,   \

That's a lot of magic numbers and formulae ... are they right? How would a user know if they are right.

To be honest you really wouldn't. It would take a lot of careful testing on numerous workloads and systems. I believe SGI is starting to do a bit of testing... I don't have the resources to do many "real world" tests.

At this stage I wouldn't let them worry you too much :P Hopefully they'll gradually improve.

Quoted 10 lines
Quoted 3 lines

Nick, can you buy off on the sched.c change? Alternatively, do you want to send that fix separately John?

I saw the ACK from Nick ... but kernel/sched.c changes will have to go through Andrew, not me. Are the ia64 and generic parts separable? If the sched.c change goes in, do other architectures need to have some equivalent change?

No, it should be ok as is. The parts should be ok to seperate if you would prefer it that way.

#4 lore raw
In reply to Jesse Barnes
Quoted 8 lines · Tony
Quoted 4 lines

Some have noticed that the overlapping sched domains code doesn't quite work as intended (it results in disjoint domains on some machines), and that a top level, machine spanning domain is needed.

Why is the solution to jam this into the ia64 specific code? From this description it sounds like a generic scheduler problem, so the solution ought to be up in some generic code.

It used to be in generic code, but now it's arch specific, so each arch builds its own scheduling domains. This patch adds some NUMA specific scheduling domain code to build a top level domain on boxes with lots of nodes. It won't affect non-NUMA or small NUMA boxes.

Quoted 9 lines

+ .min_interval = 80, \
+ .max_interval = 320, \
+ .busy_factor = 320, \
+ .imbalance_pct = 125, \
+ .cache_hot_time = (10*1000000), \
+ .balance_interval = 100*(63+num_online_cpus())/64, \

That's a lot of magic numbers and formulae ... are they right? How would a user know if they are right.

John has run several tests on large systems to come up with something reasonable, but no doubt they could use more tweaking. John or Nick, care to comment?

Quoted 7 lines
Quoted 2 lines

Nick, can you buy off on the sched.c change? Alternatively, do you want to send that fix separately John?

I saw the ACK from Nick ... but kernel/sched.c changes will have to go through Andrew, not me. Are the ia64 and generic parts separable? If the sched.c change goes in, do other architectures need to have some equivalent change?

Ok, I'll send out the sched.c bit separately then as it's standalone.

Thanks,
Jesse

#5 lore raw
In reply to Jesse Barnes

Hello Nick & all,

Quoted 14 lines · Nick Piggin
Quoted 10 lines · Tony
include/asm-ia64/topology.h +6 -0
+    .min_interval        = 80,            \
+    .max_interval        = 320,            \
+    .busy_factor        = 320,            \
+    .imbalance_pct        = 125,            \
+    .cache_hot_time        = (10*1000000),        \
+    .balance_interval    = 100*(63+num_online_cpus())/64,   \

That's a lot of magic numbers and formulae ... are they right? How would a user know if they are right.

Quoted 7 lines

To be honest you really wouldn't. It would take a lot of careful testing on numerous workloads and systems. I believe SGI is starting to do a bit of testing... I don't have the resources to do many "real world" tests.

At this stage I wouldn't let them worry you too much :P Hopefully they'll gradually improve.

Why should'nt we use the node_distance() function to build in an independant way the Numa hierarchy and compute the right parameters for each level ?

Signature · 8 lines
 Sincères salutations.
_____________________________________________________________________
 
Xavier BRU                 BULL ISD/R&D/INTEL office:     FREC B1-422
tel : +33 (0)4 76 29 77 45                    http://www-frec.bull.fr
fax : +33 (0)4 76 29 77 70                 mailto:[email protected]
addr: BULL, 1 rue de Provence, BP 208, 38432 Echirolles Cedex, FRANCE
_____________________________________________________________________
#6 lore raw
In reply to Jesse Barnes
Quoted 23 lines · Xavier Bru

Hello Nick & all,

Quoted 19 lines · Nick Piggin
Quoted 10 lines · Tony
include/asm-ia64/topology.h +6 -0
+    .min_interval        = 80,            \
+    .max_interval        = 320,            \
+    .busy_factor        = 320,            \
+    .imbalance_pct        = 125,            \
+    .cache_hot_time        = (10*1000000),        \
+    .balance_interval    = 100*(63+num_online_cpus())/64,   \

That's a lot of magic numbers and formulae ... are they right? How would a user know if they are right.

To be honest you really wouldn't. It would take a lot of careful testing on numerous workloads and systems. I believe SGI is starting to do a bit of testing... I don't have the resources to do many "real world" tests.

Quoted 9 lines
Quoted 2 lines

At this stage I wouldn't let them worry you too much :P Hopefully they'll gradually improve.

Why should'nt we use the node_distance() function to build in an independant way the Numa hierarchy and compute the right parameters for each level ?

Hi Xavier,
That would probably be a good idea where possible, although for many architectures this sort of information won't be available. It may be that we ultimately will want to represent the NUMA topology with node_distance being the first class function/measure (I personally think sched-domains should be extended into the memory topology). At the present time though, it would be a backward step to force everyone to build a node_distance table.

Two things to note - first, even if node_distance does return something meaningful, it still has to be input into a larger field of parameters, so there will still be some heuristics/fudging/tuning going on.

Second, we can do runtime probing to gain more information. For example, Ingo has a patch in the works that will compute real cache transfer times between any two CPUs, which looks promising. We can query the number of online CPUs when deciding on balancing rates, etc etc.

So in short, we basically want as much info as we can possibly gather... and that is the easy part :( People need to do tests on their real life workloads with real systems to convert this information into useful parameters.

Anyway, the scheduler isn't _quite_ at the point where you want to be doing serious fine tuning with it yet; we've got to get a few more things to go in (eg. Ingo's patch, improvements from John Hawkes, some performance patches from me, etc).

#7 lore raw
In reply to Jesse Barnes

Hi Jesse,

From: Jesse Barnes <[email protected]>
Subject: [PATCH] top level scheduler domain for ia64
Date: Tue, 19 Oct 2004 14:27:27 -0700

Quoted 8 lines

Some have noticed that the overlapping sched domains code doesn't quite work as intended (it results in disjoint domains on some machines), and that a top level, machine spanning domain is needed. This patch from John Hawkes adds it to the ia64 code. This allows processes to run on all CPUs in large systems, though balancing is limited. It should go to Linus soon now otherwise large systems will only have ~16p (depending on topology) usable by the scheduler. I sanity checked it on a small system after rediffing John's original, and he's done some testing on very large systems.

Our 32way machine still isn't configured well with the overwrapping domain partitioning. CPUs 0-15 belongs to domain (0 1 2 3 4 5 6) and CPUs 16-31 belongs to domain (0 1 4 5 6 7), which is assymmetric and at least does not reflect the real connection.

dmesg of the machine is attached.

The following patch makes the ia64 domain partitioning (maybe Altix specific ;) optional and makes the magic number (6) configurable.

What do you think of this?

include/asm-ia64/processor.h +2 -0
Index: 269-bk/include/asm-ia64/processor.h
===================================================================
@@ -20,8 +20,10 @@
 #include <asm/ptrace.h>
 #include <asm/ustack.h>
 
+#ifdef CONFIG_IA64_SCHED_SPAN
 /* Our arch specific arch_init_sched_domain is in arch/ia64/kernel/domain.c */
 #define ARCH_HAS_SCHED_DOMAIN
+#endif
 
 #define IA64_NUM_DBG_REGS	8
 /*
arch/ia64/Kconfig +18 -0
Index: 269-bk/arch/ia64/Kconfig
===================================================================
@@ -188,6 +188,24 @@
 	  or have huge holes in the physical address space for other reasons.
 	  See <file:Documentation/vm/numa> for more.
 
+config IA64_SCHED_SPAN
+	bool "Bounded spanning of scheduling domains for large NUMA"
+	depends on NUMA
+	default y if IA64_SGI_SN2
+	help
+	  Say Y to support bounded spanning of scheduling domains for large
+	  NUMA systems, which will bound the number of nodes to span
+	  for a top-level scheduling domain.  If not set, the top-level
+	  domain spans over all nodes.
+
+config IA64_SD_NODES_PER_DOMAIN
+	int "Nodes to span"
+	depends on IA64_SCHED_SPAN
+	default "6"
+	help
+	  This is the number of nodes to span for a top-level scheduling
+	  domain.
+
 config IA64_CYCLONE
 	bool "Cyclone (EXA) Time Source support"
 	help
arch/ia64/kernel/domain.c +2 -4
Index: 269-bk/arch/ia64/kernel/domain.c
===================================================================
@@ -12,8 +12,6 @@
 #include <linux/init.h>
 #include <linux/topology.h>
 
-#define SD_NODES_PER_DOMAIN 6
-
 #ifdef CONFIG_NUMA
 /**
  * find_next_best_node - find the next node to include in a sched_domain
@@ -77,7 +75,7 @@
 	cpus_or(span, span, nodemask);
 	set_bit(node, used_nodes);
 
-	for (i = 1; i < SD_NODES_PER_DOMAIN; i++) {
+	for (i = 1; i < CONFIG_IA64_SD_NODES_PER_DOMAIN; i++) {
 		int next_node = find_next_best_node(node, used_nodes);
 		nodemask = node_to_cpumask(next_node);
 		cpus_or(span, span, nodemask);
@@ -158,7 +156,7 @@
 
 #ifdef CONFIG_NUMA
 		if (num_online_cpus()
-				> SD_NODES_PER_DOMAIN*cpus_weight(nodemask)) {
+		    > CONFIG_IA64_SD_NODES_PER_DOMAIN*cpus_weight(nodemask)) {
 			sd = &per_cpu(allnodes_domains, i);
 			*sd = SD_ALLNODES_INIT;
 			sd->span = cpu_default_map;
arch/ia64/kernel/Makefile +2 -1
Index: 269-bk/arch/ia64/kernel/Makefile
===================================================================
@@ -14,7 +14,8 @@
 obj-$(CONFIG_IA64_PALINFO)	+= palinfo.o
 obj-$(CONFIG_IOSAPIC)		+= iosapic.o
 obj-$(CONFIG_MODULES)		+= module.o
-obj-$(CONFIG_SMP)		+= smp.o smpboot.o domain.o
+obj-$(CONFIG_SMP)		+= smp.o smpboot.o
+obj-$(CONFIG_IA64_SCHED_SPAN)	+= domain.o
 obj-$(CONFIG_PERFMON)		+= perfmon_default_smpl.o
 obj-$(CONFIG_IA64_CYCLONE)	+= cyclone.o
 obj-$(CONFIG_IA64_MCA_RECOVERY)	+= mca_recovery.o

---
Takayoshi Kochi

Brought up 32 CPUs
Total of 32 processors activated (47813.16 BogoMIPS). CPU0:
domain 0: span 0000000f
groups: 00000001 00000002 00000004 00000008
domain 1: span 00ffffff
groups: 0000000f 000000f0 00000f00 0000f000 000f0000 00f00000 domain 2: span ffffffff

    groups: 0000000f 000000f0 00000f00 0000f000 000f0000 00f00000 0f000000 f0000000

CPU1:
domain 0: span 0000000f
groups: 00000002 00000004 00000008 00000001
domain 1: span 00ffffff
groups: 0000000f 000000f0 00000f00 0000f000 000f0000 00f00000 domain 2: span ffffffff

    groups: 0000000f 000000f0 00000f00 0000f000 000f0000 00f00000 0f000000 f0000000

CPU2:
domain 0: span 0000000f
groups: 00000004 00000008 00000001 00000002
domain 1: span 00ffffff
groups: 0000000f 000000f0 00000f00 0000f000 000f0000 00f00000 domain 2: span ffffffff

    groups: 0000000f 000000f0 00000f00 0000f000 000f0000 00f00000 0f000000 f0000000

CPU3:
domain 0: span 0000000f
groups: 00000008 00000001 00000002 00000004
domain 1: span 00ffffff
groups: 0000000f 000000f0 00000f00 0000f000 000f0000 00f00000 domain 2: span ffffffff

    groups: 0000000f 000000f0 00000f00 0000f000 000f0000 00f00000 0f000000 f0000000

CPU4:
domain 0: span 000000f0
groups: 00000010 00000020 00000040 00000080
domain 1: span 00ffffff
groups: 000000f0 00000f00 0000f000 000f0000 00f00000 0000000f domain 2: span ffffffff

    groups: 000000f0 00000f00 0000f000 000f0000 00f00000 0f000000 f0000000 0000000f

CPU5:
domain 0: span 000000f0
groups: 00000020 00000040 00000080 00000010
domain 1: span 00ffffff
groups: 000000f0 00000f00 0000f000 000f0000 00f00000 0000000f domain 2: span ffffffff

    groups: 000000f0 00000f00 0000f000 000f0000 00f00000 0f000000 f0000000 0000000f

CPU6:
domain 0: span 000000f0
groups: 00000040 00000080 00000010 00000020
domain 1: span 00ffffff
groups: 000000f0 00000f00 0000f000 000f0000 00f00000 0000000f domain 2: span ffffffff

    groups: 000000f0 00000f00 0000f000 000f0000 00f00000 0f000000 f0000000 0000000f

CPU7:
domain 0: span 000000f0
groups: 00000080 00000010 00000020 00000040
domain 1: span 00ffffff
groups: 000000f0 00000f00 0000f000 000f0000 00f00000 0000000f domain 2: span ffffffff

    groups: 000000f0 00000f00 0000f000 000f0000 00f00000 0f000000 f0000000 0000000f

CPU8:
domain 0: span 00000f00
groups: 00000100 00000200 00000400 00000800
domain 1: span 00ffffff
groups: 00000f00 0000f000 000f0000 00f00000 0000000f 000000f0 domain 2: span ffffffff

    groups: 00000f00 0000f000 000f0000 00f00000 0f000000 f0000000 0000000f 000000f0

CPU9:
domain 0: span 00000f00
groups: 00000200 00000400 00000800 00000100
domain 1: span 00ffffff
groups: 00000f00 0000f000 000f0000 00f00000 0000000f 000000f0 domain 2: span ffffffff

    groups: 00000f00 0000f000 000f0000 00f00000 0f000000 f0000000 0000000f 000000f0

CPU10:
domain 0: span 00000f00
groups: 00000400 00000800 00000100 00000200
domain 1: span 00ffffff
groups: 00000f00 0000f000 000f0000 00f00000 0000000f 000000f0 domain 2: span ffffffff

    groups: 00000f00 0000f000 000f0000 00f00000 0f000000 f0000000 0000000f 000000f0

CPU11:
domain 0: span 00000f00
groups: 00000800 00000100 00000200 00000400
domain 1: span 00ffffff
groups: 00000f00 0000f000 000f0000 00f00000 0000000f 000000f0 domain 2: span ffffffff

    groups: 00000f00 0000f000 000f0000 00f00000 0f000000 f0000000 0000000f 000000f0

CPU12:
domain 0: span 0000f000
groups: 00001000 00002000 00004000 00008000
domain 1: span 00ffffff
groups: 0000f000 000f0000 00f00000 0000000f 000000f0 00000f00 domain 2: span ffffffff

    groups: 0000f000 000f0000 00f00000 0f000000 f0000000 0000000f 000000f0 00000f00

CPU13:
domain 0: span 0000f000
groups: 00002000 00004000 00008000 00001000
domain 1: span 00ffffff
groups: 0000f000 000f0000 00f00000 0000000f 000000f0 00000f00 domain 2: span ffffffff

    groups: 0000f000 000f0000 00f00000 0f000000 f0000000 0000000f 000000f0 00000f00

CPU14:
domain 0: span 0000f000
groups: 00004000 00008000 00001000 00002000
domain 1: span 00ffffff
groups: 0000f000 000f0000 00f00000 0000000f 000000f0 00000f00 domain 2: span ffffffff

    groups: 0000f000 000f0000 00f00000 0f000000 f0000000 0000000f 000000f0 00000f00

CPU15:
domain 0: span 0000f000
groups: 00008000 00001000 00002000 00004000
domain 1: span 00ffffff
groups: 0000f000 000f0000 00f00000 0000000f 000000f0 00000f00 domain 2: span ffffffff

    groups: 0000f000 000f0000 00f00000 0f000000 f0000000 0000000f 000000f0 00000f00

CPU16:
domain 0: span 000f0000
groups: 00010000 00020000 00040000 00080000
domain 1: span ffff00ff
groups: 000f0000 00f00000 0f000000 f0000000 0000000f 000000f0 domain 2: span ffffffff

    groups: 000f0000 00f00000 0f000000 f0000000 0000000f 000000f0 00000f00 0000f000

CPU17:
domain 0: span 000f0000
groups: 00020000 00040000 00080000 00010000
domain 1: span ffff00ff
groups: 000f0000 00f00000 0f000000 f0000000 0000000f 000000f0 domain 2: span ffffffff

    groups: 000f0000 00f00000 0f000000 f0000000 0000000f 000000f0 00000f00 0000f000

CPU18:
domain 0: span 000f0000
groups: 00040000 00080000 00010000 00020000
domain 1: span ffff00ff
groups: 000f0000 00f00000 0f000000 f0000000 0000000f 000000f0 domain 2: span ffffffff

    groups: 000f0000 00f00000 0f000000 f0000000 0000000f 000000f0 00000f00 0000f000

CPU19:
domain 0: span 000f0000
groups: 00080000 00010000 00020000 00040000
domain 1: span ffff00ff
groups: 000f0000 00f00000 0f000000 f0000000 0000000f 000000f0 domain 2: span ffffffff

    groups: 000f0000 00f00000 0f000000 f0000000 0000000f 000000f0 00000f00 0000f000

CPU20:
domain 0: span 00f00000
groups: 00100000 00200000 00400000 00800000
domain 1: span ffff00ff
groups: 00f00000 0f000000 f0000000 0000000f 000000f0 000f0000 domain 2: span ffffffff

    groups: 00f00000 0f000000 f0000000 0000000f 000000f0 00000f00 0000f000 000f0000

CPU21:
domain 0: span 00f00000
groups: 00200000 00400000 00800000 00100000
domain 1: span ffff00ff
groups: 00f00000 0f000000 f0000000 0000000f 000000f0 000f0000 domain 2: span ffffffff

    groups: 00f00000 0f000000 f0000000 0000000f 000000f0 00000f00 0000f000 000f0000

CPU22:
domain 0: span 00f00000
groups: 00400000 00800000 00100000 00200000
domain 1: span ffff00ff
groups: 00f00000 0f000000 f0000000 0000000f 000000f0 000f0000 domain 2: span ffffffff

    groups: 00f00000 0f000000 f0000000 0000000f 000000f0 00000f00 0000f000 000f0000

CPU23:
domain 0: span 00f00000
groups: 00800000 00100000 00200000 00400000
domain 1: span ffff00ff
groups: 00f00000 0f000000 f0000000 0000000f 000000f0 000f0000 domain 2: span ffffffff

    groups: 00f00000 0f000000 f0000000 0000000f 000000f0 00000f00 0000f000 000f0000

CPU24:
domain 0: span 0f000000
groups: 01000000 02000000 04000000 08000000
domain 1: span ffff00ff
groups: 0f000000 f0000000 0000000f 000000f0 000f0000 00f00000 domain 2: span ffffffff

    groups: 0f000000 f0000000 0000000f 000000f0 00000f00 0000f000 000f0000 00f00000

CPU25:
domain 0: span 0f000000
groups: 02000000 04000000 08000000 01000000
domain 1: span ffff00ff
groups: 0f000000 f0000000 0000000f 000000f0 000f0000 00f00000 domain 2: span ffffffff

    groups: 0f000000 f0000000 0000000f 000000f0 00000f00 0000f000 000f0000 00f00000

CPU26:
domain 0: span 0f000000
groups: 04000000 08000000 01000000 02000000
domain 1: span ffff00ff
groups: 0f000000 f0000000 0000000f 000000f0 000f0000 00f00000 domain 2: span ffffffff

    groups: 0f000000 f0000000 0000000f 000000f0 00000f00 0000f000 000f0000 00f00000

CPU27:
domain 0: span 0f000000
groups: 08000000 01000000 02000000 04000000
domain 1: span ffff00ff
groups: 0f000000 f0000000 0000000f 000000f0 000f0000 00f00000 domain 2: span ffffffff

    groups: 0f000000 f0000000 0000000f 000000f0 00000f00 0000f000 000f0000 00f00000

CPU28:
domain 0: span f0000000
groups: 10000000 20000000 40000000 80000000
domain 1: span ffff00ff
groups: f0000000 0000000f 000000f0 000f0000 00f00000 0f000000 domain 2: span ffffffff

    groups: f0000000 0000000f 000000f0 00000f00 0000f000 000f0000 00f00000 0f000000

CPU29:
domain 0: span f0000000
groups: 20000000 40000000 80000000 10000000
domain 1: span ffff00ff
groups: f0000000 0000000f 000000f0 000f0000 00f00000 0f000000 domain 2: span ffffffff

    groups: f0000000 0000000f 000000f0 00000f00 0000f000 000f0000 00f00000 0f000000

CPU30:
domain 0: span f0000000
groups: 40000000 80000000 10000000 20000000
domain 1: span ffff00ff
groups: f0000000 0000000f 000000f0 000f0000 00f00000 0f000000 domain 2: span ffffffff

    groups: f0000000 0000000f 000000f0 00000f00 0000f000 000f0000 00f00000 0f000000

CPU31:
domain 0: span f0000000
groups: 80000000 10000000 20000000 40000000
domain 1: span ffff00ff
groups: f0000000 0000000f 000000f0 000f0000 00f00000 0f000000 domain 2: span ffffffff

    groups: f0000000 0000000f 000000f0 00000f00 0000f000 000f0000 00f00000 0f000000
#8 lore raw
In reply to Jesse Barnes
Quoted 4 lines · Takayoshi Kochi

Our 32way machine still isn't configured well with the overwrapping domain partitioning. CPUs 0-15 belongs to domain (0 1 2 3 4 5 6) and CPUs 16-31 belongs to domain (0 1 4 5 6 7), which is assymmetric and at least does not reflect the real connection.

Hm, I was afraid of that since the top level domain is only built if the system has more CPUs than SD_NODES_PER_DOMAIN * cpus_per_node. We could change that to be a simple if (numnodes > SD_NODES_PER_DOMAIN) instead.

Quoted 6 lines

dmesg of the machine is attached.

The following patch makes the ia64 domain partitioning (maybe Altix specific ;) optional and makes the magic number (6) configurable.

What do you think of this?

Maybe a boot parameter would be better for configuring SD_NODES_PER_DOMAIN? That would allow a single kernel binary to be configured to run well on many types of NUMA systems. It would also mean you could play with very large node domains, with and w/o a top level domain.

Jesse

#9 lore raw
In reply to Jesse Barnes

Hi Jesse,

From: Jesse Barnes <[email protected]>
Subject: Re: [PATCH] top level scheduler domain for ia64
Date: Thu, 28 Oct 2004 08:26:00 -0700

Quoted 8 lines
Quoted 4 lines

Our 32way machine still isn't configured well with the overwrapping domain partitioning. CPUs 0-15 belongs to domain (0 1 2 3 4 5 6) and CPUs 16-31 belongs to domain (0 1 4 5 6 7), which is assymmetric and at least does not reflect the real connection.

Hm, I was afraid of that since the top level domain is only built if the system has more CPUs than SD_NODES_PER_DOMAIN * cpus_per_node. We could change that to be a simple if (numnodes > SD_NODES_PER_DOMAIN) instead.

But I don't think the domain partitioning thing will optimize anything in all the cases that (numnodes > SD_NODES_PER_DOMAIN) is true.

Quoted 11 lines
Quoted 6 lines

dmesg of the machine is attached.

The following patch makes the ia64 domain partitioning (maybe Altix specific ;) optional and makes the magic number (6) configurable.

What do you think of this?

Maybe a boot parameter would be better for configuring SD_NODES_PER_DOMAIN? That would allow a single kernel binary to be configured to run well on many types of NUMA systems. It would also mean you could play with very large node domains, with and w/o a top level domain.

It sounds good for -mm, but not for Linus?
That would be a pain for any ordinary user to test many parameters for each configuration (and the number makes very little difference, which would make it more difficult to find the best number).

What I'd like to do now is to get rid of subarch specific optimization out of generic kernels.

Do you have any objection for the patch to be included in the mainline?

---
Takayoshi Kochi

#10 lore raw
In reply to Jesse Barnes
Quoted 11 lines · Takayoshi Kochi
Quoted 5 lines

Maybe a boot parameter would be better for configuring SD_NODES_PER_DOMAIN? That would allow a single kernel binary to be configured to run well on many types of NUMA systems. It would also mean you could play with very large node domains, with and w/o a top level domain.

It sounds good for -mm, but not for Linus?
That would be a pain for any ordinary user to test many parameters for each configuration (and the number makes very little difference, which would make it more difficult to find the best number).

Quoted 4 lines

What I'd like to do now is to get rid of subarch specific optimization out of generic kernels.

Do you have any objection for the patch to be included in the mainline?

I don't want it included since it hurts generic kernels on large systems (i.e. either they wouldn't boot or wouldn't see all the CPUs). It should be possible to use a runtime check (whether a boot parameter or just a comparison on numnodes) instead to avoid non-overlapping domains.

If I understand you right, you don't want a top level domain for your 32 way systems, but you *do* want the node domains to span the whole thing. Is that right?

If so, you could do something like this I think?

if (numnodes <= SMALL_SYSTEM_THRESHOLD) {
SD_NODES_PER_DOMAIN = numnodes;
build_node_domains(); /* each one spans the system */
} else {
SD_NODES_PER_DOMAIN = 4; /* or whatever */ build_node_domains(); /* only spans nearby nodes */ build_top_level_domain(); /* whole system, infrequently balanced */
}

Would that address your concerns?

Jesse

#11 lore raw
In reply to Jesse Barnes
Quoted 15 lines · Jesse Barnes

If I understand you right, you don't want a top level domain for your 32 way systems, but you *do* want the node domains to span the whole thing. Is that right?

If so, you could do something like this I think?

if (numnodes <= SMALL_SYSTEM_THRESHOLD) {
SD_NODES_PER_DOMAIN = numnodes;
build_node_domains(); /* each one spans the system */
} else {
SD_NODES_PER_DOMAIN = 4; /* or whatever */ build_node_domains(); /* only spans nearby nodes */ build_top_level_domain(); /* whole system, infrequently balanced */
}

Quoted 1 line

Would that address your concerns?

Doesn't sound like a great idea. HP's already shipping 128-way Superdome IA-64 systems, and they'll want to be set up rather differently from the Altix systems. I think this code needs to be autotuning so it doesn't need to be touched whenever a vendor releases a new configuration (I think I heard that NASA's Altixes had a custom CPU brick with twice the CPUs in it?)

Signature · 6 lines
"Next the statesmen will invent cheap lies, putting the blame upon 
the nation that is attacked, and every man will be glad of those
conscience-soothing falsities, and will diligently study them, and refuse
to examine any refutations of them; and thus he will by and by convince 
himself that the war is just, and will thank God for the better sleep 
he enjoys after this process of grotesque self-deception." -- Mark Twain
#12 lore raw
In reply to Jesse Barnes
Quoted 16 lines · Matthew Wilcox
Quoted 15 lines · Jesse Barnes

If I understand you right, you don't want a top level domain for your 32 way systems, but you *do* want the node domains to span the whole thing. Is that right?

If so, you could do something like this I think?

if (numnodes <= SMALL_SYSTEM_THRESHOLD) {
SD_NODES_PER_DOMAIN = numnodes;
build_node_domains(); /* each one spans the system */
} else {
SD_NODES_PER_DOMAIN = 4; /* or whatever */ build_node_domains(); /* only spans nearby nodes */ build_top_level_domain(); /* whole system, infrequently balanced */
}

Quoted 8 lines
Quoted 1 line

Would that address your concerns?

Doesn't sound like a great idea. HP's already shipping 128-way Superdome IA-64 systems, and they'll want to be set up rather differently from the Altix systems. I think this code needs to be autotuning so it doesn't need to be touched whenever a vendor releases a new configuration (I think I heard that NASA's Altixes had a custom CPU brick with twice the CPUs in it?)

Yeah, but still the same number of CPUs/FSB. I agree that autotuning would be best (the above is a crude example of that). Any suggestions?

Jesse

#13 lore raw
In reply to Jesse Barnes
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

if (numnodes <= SMALL_SYSTEM_THRESHOLD) {
SD_NODES_PER_DOMAIN = numnodes; build_node_domains(); /* each one spans the system */
} else {
SD_NODES_PER_DOMAIN = 4; /* or whatever */

Ugh! The magic '4' is going to be different for different machines, so there is no value that will make all of the people happy all of the time.

Quoted 3 lines

Yeah, but still the same number of CPUs/FSB. I agree that autotuning would be
best (the above is a crude example of that). Any suggestions?

Can't you use the SLIT table to build a hierachy of scheduler domains that matches the actual hardware configuration, instead of using compiled in constants or even boot-time command line parameters?

-Tony

#14 lore raw
In reply to Jesse Barnes
Quoted 9 lines · Tony
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

if (numnodes <= SMALL_SYSTEM_THRESHOLD) {
SD_NODES_PER_DOMAIN = numnodes; build_node_domains(); /* each one spans the system */
} else {
SD_NODES_PER_DOMAIN = 4; /* or whatever */

Ugh! The magic '4' is going to be different for different machines, so there is no value that will make all of the people happy all of the time.

Right.

Quoted 8 lines
Quoted 3 lines

Yeah, but still the same number of CPUs/FSB. I agree that autotuning would be
best (the above is a crude example of that). Any suggestions?

Can't you use the SLIT table to build a hierachy of scheduler domains that matches the actual hardware configuration, instead of using compiled in constants or even boot-time command line parameters?

Maybe, but that will get complex very quickly I think. Right now we have three domains on ia64, the cpu domain, the node domain, which contains several nodes worth of CPUs, and the top level domain, which spans the whole machine.

So there are two questions, how big should the node domain be and should it span the whole machine (avoiding the need for a top level domain)? Obviously the answer is pretty machine specific, and I'm not sure the SLIT helps us much since its values are arbitrary distance values, not anything concrete.

Jesse

#15 lore raw
In reply to Jesse Barnes
Quoted 9 lines

Maybe, but that will get complex very quickly I think. Right now we have three domains on ia64, the cpu domain, the node domain, which contains several nodes worth of CPUs, and the top level domain, which spans the whole machine.

So there are two questions, how big should the node domain be and should it span the whole machine (avoiding the need for a top level domain)? Obviously the answer is pretty machine specific, and I'm not sure the SLIT helps us much since its values are arbitrary distance values, not anything concrete.

I'd pose a broader question ... are the manufacturers of big machines happy with three domains? Perhaps it makes sense to allow for more levels that match the physical parameters of the machine. E.g. the NEC box has 4 cpus per-node, and 4 nodes in a "super-node", and 2 "super-nodes" in a machine. It would make sense to me if there was a scheduler domain level that would handle balancing between the nodes in a super-node in addition to the top-level domain to handle balancing between super-nodes.

While the values in SLIT can be somewhat abstract, they could be used to derive the whole node, super-node, hyper-node, ultra-node, marketting-zeta-node structure to build as many levels as make sense into the scheduler.

Or am I over-engineering?

-Tony

#16 lore raw
In reply to Jesse Barnes
Quoted 8 lines · Tony

I'd pose a broader question ... are the manufacturers of big machines happy with three domains? Perhaps it makes sense to allow for more levels that match the physical parameters of the machine. E.g. the NEC box has 4 cpus per-node, and 4 nodes in a "super-node", and 2 "super-nodes" in a machine. It would make sense to me if there was a scheduler domain level that would handle balancing between the nodes in a super-node in addition to the top-level domain to handle balancing between super-nodes.

Quoted 6 lines

While the values in SLIT can be somewhat abstract, they could be used to derive the whole node, super-node, hyper-node, ultra-node, marketting-zeta-node structure to build as many levels as make sense into the scheduler.

Or am I over-engineering?

Just guessing, but I think that might be overkill. We'd have to collect data to know for sure though (which means someone has to implement it :).

Jesse

#17 lore raw
In reply to Jesse Barnes

I have a patch in the latest mm tree which allow user to adjust scheduler domain parameters via sysctl.
However I have not include SPAN in the controllable parameters list. Would including SPAN in sysctl parameter list helpful to this issue?

Zou Nan hai

-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of Jesse Barnes
Sent: Tuesday, November 02, 2004 3:02 AM
To: Luck, Tony
Cc: Matthew Wilcox; Takayoshi Kochi; [email protected]; [email protected]; [email protected]
Subject: Re: [PATCH] top level scheduler domain for ia64

Quoted 9 lines · Tony
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

if (numnodes <= SMALL_SYSTEM_THRESHOLD) {
SD_NODES_PER_DOMAIN = numnodes; build_node_domains(); /* each one spans the system */
} else {
SD_NODES_PER_DOMAIN = 4; /* or whatever */

Ugh! The magic '4' is going to be different for different machines, so there is no value that will make all of the people happy all of the time.

Right.

Quoted 8 lines
Quoted 3 lines

Yeah, but still the same number of CPUs/FSB. I agree that autotuning would be
best (the above is a crude example of that). Any suggestions?

Can't you use the SLIT table to build a hierachy of scheduler domains that matches the actual hardware configuration, instead of using compiled in constants or even boot-time command line parameters?

Maybe, but that will get complex very quickly I think. Right now we have
three domains on ia64, the cpu domain, the node domain, which contains several nodes worth of CPUs, and the top level domain, which spans the whole
machine.

So there are two questions, how big should the node domain be and should it
span the whole machine (avoiding the need for a top level domain)? Obviously
the answer is pretty machine specific, and I'm not sure the SLIT helps us
much since its values are arbitrary distance values, not anything concrete.

Jesse
-
To unsubscribe from this list: send the line "unsubscribe linux-ia64" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#18 lore raw
In reply to Jesse Barnes

Hi,

From: "Luck, Tony" <[email protected]>
Subject: RE: [PATCH] top level scheduler domain for ia64
Date: Mon, 1 Nov 2004 11:45:44 -0800

Quoted 9 lines
Quoted 9 lines

Maybe, but that will get complex very quickly I think. Right now we have three domains on ia64, the cpu domain, the node domain, which contains several nodes worth of CPUs, and the top level domain, which spans the whole machine.

So there are two questions, how big should the node domain be and should it span the whole machine (avoiding the need for a top level domain)? Obviously the answer is pretty machine specific, and I'm not sure the SLIT helps us much since its values are arbitrary distance values, not anything concrete.

Right, but if SLIT is implemented in real firmware (not a fake), the numbers should have some meanings to describe how the machine is configured.

For machines which have tree-like hierarchy NUMA, the SLIT table can help enough to construct scheduling domains (like NEC's, and I suppose Superdome is similar) and fits well for the sched-domain scheduler.

The problem is that there can be other forms of topology. It may be hard to derive its actual connection only with SLIT. Quoting a SLIT example from the ACPI spec 3.0...

domain [0] [1] [2] [3]

   [0]  10  15  20  18
   [1]  15  10  16  24
   [2]  20  16  10  12
   [3]  18  24  12  10

This is an example which is not real, but it's possible from the viewpoint of the ACPI spec;)
For this case, it's very hard to say it's worth making other 3 levels of scheduling domains above the cpu domain.

The userspace adjustment which Nanhai proposed does not work for booting the 512-way Altix anyway (if we remove the node domain from the generic kernel).

I think the short-term solution is to make the boot-time parameter like Jesse said to control the creation and span size of the node domain and see what is necessary next.

---
Takayoshi Kochi

#19 lore raw
In reply to Jesse Barnes
Quoted 6 lines · Matthew Wilcox

Doesn't sound like a great idea. HP's already shipping 128-way Superdome IA-64 systems, and they'll want to be set up rather differently from the Altix systems. I think this code needs to be autotuning so it doesn't need to be touched whenever a vendor releases a new configuration (I think I heard that NASA's Altixes had a custom CPU brick with twice the CPUs in it?)

Yup. Though that new brick is not specific to NASA's new Columbia system.

http://www.sgi.com/company_info/newsroom/press_releases/2004/october/altix_bx2.html

| SGI Doubles Density of High-End Altix Server
|
| New Altix 3700 Bx2 Ratchets Up Performance and Cost-Efficiency with 
| Twice the Bandwidth, Half the Footprint of Flagship System 
|
| MOUNTAIN VIEW, Calif., (October 28, 2004) [...] Silicon Graphics today 
| unveiled a new version of its SGI® Altix® 3700 system that delivers 
| twice the bandwidth and processor density of its flagship high-end 
| model.

Gerald (not working for SGI ;-)

#20 lore raw
In reply to Jesse Barnes

From: Takayoshi Kochi <[email protected]>
Subject: Re: [PATCH] top level scheduler domain for ia64
Date: Tue, 02 Nov 2004 16:36:36 +0900 (JST)

Quoted 3 lines

I think the short-term solution is to make the boot-time parameter like Jesse said to control the creation and span size of the node domain and see what is necessary next.

This patch adds a new kernel parameter "nodes_per_domain" which specifies how many nodes are included in a node scheduling domain.

Index: 269-bk/arch/ia64/kernel/domain.c
=================================--- 269-bk/arch/ia64/kernel/domain.c (revision 49)
+++ 269-bk/arch/ia64/kernel/domain.c (working copy)
@@ -12,9 +12,9 @@
#include <linux/init.h>
#include <linux/topology.h>

-#define SD_NODES_PER_DOMAIN 6
+#ifdef CONFIG_NUMA
+static int sd_nodes_per_domain = 6;

-#ifdef CONFIG_NUMA
/**
* find_next_best_node - find the next node to include in a sched_domain
* @node: node whose sched_domain we're building
@@ -77,7 +77,7 @@
cpus_or(span, span, nodemask);
set_bit(node, used_nodes);

- for (i = 1; i < SD_NODES_PER_DOMAIN; i++) {
+ for (i = 1; i < sd_nodes_per_domain; i++) {
int next_node = find_next_best_node(node, used_nodes); nodemask = node_to_cpumask(next_node); cpus_or(span, span, nodemask);
@@ -158,7 +158,7 @@

#ifdef CONFIG_NUMA
if (num_online_cpus()
- > SD_NODES_PER_DOMAIN*cpus_weight(nodemask)) {
+ > sd_nodes_per_domain*cpus_weight(nodemask)) {
sd = &per_cpu(allnodes_domains, i); *sd = SD_ALLNODES_INIT; sd->span = cpu_default_map;
@@ -379,3 +379,18 @@
#endif
}

+#ifdef CONFIG_NUMA
+static int __init
+set_nodes_per_domain (char *str)
+{
+ int tmp;
+
+ get_option(&str, &tmp);
+ if (tmp > 1 && tmp <= MAX_NUMNODES)
+ sd_nodes_per_domain = tmp;
+
+ return 1;
+}
+
+__setup("nodes_per_domain=", set_nodes_per_domain); +#endif

---
Takayoshi Kochi

#21 lore raw
In reply to Jesse Barnes
Quoted 7 lines
Quoted 3 lines

I think the short-term solution is to make the boot-time parameter like Jesse said to control the creation and span size of the node domain and see what is necessary next.

This patch adds a new kernel parameter "nodes_per_domain" which specifies how many nodes are included in a node scheduling domain.

Can you add a short write-up for Documentation/kernel-parameters.txt to explain the effect, and how a user should pick what value to use (examples would be good ... if SGI want to throw in some values for small/large Altix boxes, that would be good too).

Is '6' a reasonable default if the parameter is not set?

-Tony

#22 lore raw
In reply to Jesse Barnes

Hi,

From: "Luck, Tony" <[email protected]>
Subject: RE: [PATCH] top level scheduler domain for ia64
Date: Tue, 2 Nov 2004 13:31:16 -0800

Quoted 4 lines
Quoted 4 lines
Quoted 3 lines

I think the short-term solution is to make the boot-time parameter like Jesse said to control the creation and span size of the node domain and see what is necessary next.

Quoted 8 lines
Quoted 3 lines

This patch adds a new kernel parameter "nodes_per_domain" which specifies how many nodes are included in a node scheduling domain.

Can you add a short write-up for Documentation/kernel-parameters.txt to explain the effect, and how a user should pick what value to use (examples would be good ... if SGI want to throw in some values for small/large Altix boxes, that would be good too).

The diff is attached.
Though I'm not sure how to pick the best number... Any ideas from SGI?

Quoted 1 line

Is '6' a reasonable default if the parameter is not set?

Maybe for Altix. I guess the only user of this domain split is Altix, so it's ok ;)

The patch below is a documentation for 'nodes_per_domain=' option.

=== my-bk/Documentation/kernel-parameters.txt 1.57 vs edited ==--- 1.57/Documentation/kernel-parameters.txt 2004-10-30 18:31:21 +09:00
+++ edited/my-bk/Documentation/kernel-parameters.txt 2004-11-03 15:07:20 +09:00
@@ -475,6 +475,15 @@

hugepages= [HW,IA-32,IA-64] Maximal number of HugeTLB pages.

+ nodes_per_domain=n
+ [IA-64] specifies number of nodes in a scheduling
+ domain. If the number of nodes in a sysmtem is
+ large enough, the node scheduling domain (usually
+ spans the whole system) is split into <n> nodes
+ for each. Default <n> is 6.
+ If you have a large NUMA machine, consider using
+ this option to reduce load balancing overhead.
+
noirqbalance [IA-32,SMP,KNL] Disable kernel irq balancing

i8042.direct [HW] Put keyboard port into non-translated mode

---
Takayoshi Kochi

#23 lore raw
In reply to Jesse Barnes
Quoted 19 lines · Takayoshi Kochi

Hi,

From: "Luck, Tony" <[email protected]>
Subject: RE: [PATCH] top level scheduler domain for ia64
Date: Tue, 2 Nov 2004 13:31:16 -0800

Quoted 4 lines
Quoted 4 lines
Quoted 3 lines

I think the short-term solution is to make the boot-time parameter like Jesse said to control the creation and span size of the node domain and see what is necessary next.

Quoted 8 lines
Quoted 3 lines

This patch adds a new kernel parameter "nodes_per_domain" which specifies how many nodes are included in a node scheduling domain.

Can you add a short write-up for Documentation/kernel-parameters.txt to explain the effect, and how a user should pick what value to use (examples would be good ... if SGI want to throw in some values for small/large Altix boxes, that would be good too).

Quoted 8 lines

The diff is attached.
Though I'm not sure how to pick the best number... Any ideas from SGI?

Quoted 1 line

Is '6' a reasonable default if the parameter is not set?

Maybe for Altix. I guess the only user of this domain split is Altix, so it's ok ;)

Yep, looks good to me. Thanks.

Jesse

#24 lore raw
In reply to Jesse Barnes
Quoted 6 lines
Quoted 4 lines
Quoted 1 line

Is '6' a reasonable default if the parameter is not set?

Maybe for Altix. I guess the only user of this domain split is Altix, so it's ok ;)

Yep, looks good to me. Thanks.

So tell me who wants a value other than 6, and what is the value that they want? I was hoping for some more specific advice in the kernel-parameters.txt file. What's a good choice for a 16-node machine? For a 128-node machine?

-Tony

#25 lore raw
In reply to Jesse Barnes
Quoted 11 lines · Tony
Quoted 6 lines
Quoted 4 lines
Quoted 1 line

Is '6' a reasonable default if the parameter is not set?

Maybe for Altix. I guess the only user of this domain split is Altix, so it's ok ;)

Yep, looks good to me. Thanks.

So tell me who wants a value other than 6, and what is the value that they want? I was hoping for some more specific advice in the kernel-parameters.txt file. What's a good choice for a 16-node machine? For a 128-node machine?

I don't know. The original value I chose was 4, but I think Nick changed it to 6 to get more overlap. Any ideas John?

Jesse

#26 lore raw
In reply to Jesse Barnes

From: "Jesse Barnes" <[email protected]>

Quoted 7 lines
Quoted 4 lines

So tell me who wants a value other than 6, and what is the value that they want? I was hoping for some more specific advice in the kernel-parameters.txt file. What's a good choice for a 16-node machine? For a 128-node machine?

I don't know. The original value I chose was 4, but I think Nick changed it to 6 to get more overlap. Any ideas John?

I haven't been able to get enough reliably reproducible performance numbers using the recent -mm kernels to allow me to experiment with different nodes-per-domain values. Using 6 and having an occasionally load-balancing umbrella covering all the CPUs in the system seems to behave reasonably for 64p and above. I just haven't tried other values, other than once trying 32 and without that all-CPU umbrella, and that performance was bad.

John Hawkes