Skip to contentSkip to conversation

random ia64 sched-domains values

PATCH In review Nick Piggin 2 participants 6 messages Feb 25, 2005 – Feb 26, 2005

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 random ia64 sched-domains values

Hi,
These are some values I just found from thin-air that should hopefully make ia64 work after the latest sched patches.

Though it did look like there may have been some problems with the old values, which I tried to fix up too:

- SD_ALLNODES_INIT had .balance_interval set a specific way - this field is internal to the scheduler and will end up sitting between min_interval and max_interval and not do what was intended, so tweak max_interval instead.

- SD_ALLNODES_INIT had SD_BALANCE_EXEC set. The scheduler doesn't cope _really_ well with doing balance-on-exec over the top level of a multi-level NUMA setup like this. It will tend to fill up the first CPUs in each high level group (cluster of 6 nodes) rather than spreading the load on a per-node basis. This could be improved if we really need it, but for now remove the flag from the top level.

- Increase SD_NODES_PER_DOMAIN in the low level NUMA domain so we still push tasks out a reasonable distance. Set SD_BALANCE_FORK in this domain too.

Not compiled or tested, sorry.

---

 linux-2.6-npiggin/arch/ia64/kernel/domain.c   |    2 
 linux-2.6-npiggin/include/asm-ia64/topology.h |   62 ++++++++++++++++++++------
 2 files changed, 49 insertions(+), 15 deletions(-)
include/asm-ia64/topology.h renamed from include/asm-ia64/topology.h~sched-tune-ia64 +48 -14
diff -puN include/asm-ia64/topology.h~sched-tune-ia64 include/asm-ia64/topology.h
@@ -42,25 +42,55 @@
 
 void build_cpu_to_node_map(void);
 
+#define SD_CPU_INIT (struct sched_domain) {		\
+	.span			= CPU_MASK_NONE,	\
+	.parent			= NULL,			\
+	.groups			= NULL,			\
+	.min_interval		= 1,			\
+	.max_interval		= 4,			\
+	.busy_factor		= 64,			\
+	.imbalance_pct		= 125,			\
+	.cache_hot_time		= (10*1000000),		\
+	.cache_nice_tries	= 2,			\
+	.per_cpu_gain		= 100,			\
+	.cache_nice_tries	= 2,			\
+	.busy_idx		= 2,			\
+	.idle_idx		= 1,			\
+	.newidle_idx		= 2,			\
+	.wake_idx		= 1,			\
+	.forkexec_idx		= 1,			\
+	.flags			= SD_LOAD_BALANCE	\
+				| SD_BALANCE_NEWIDLE	\
+				| SD_BALANCE_EXEC	\
+				| SD_WAKE_AFFINE,	\
+	.last_balance		= jiffies,		\
+	.balance_interval	= 1,			\
+	.nr_balance_failed	= 0,			\
+}
+
 /* sched_domains SD_NODE_INIT for IA64 NUMA machines */
 #define SD_NODE_INIT (struct sched_domain) {		\
 	.span			= CPU_MASK_NONE,	\
 	.parent			= NULL,			\
 	.groups			= NULL,			\
-	.min_interval		= 80,			\
-	.max_interval		= 320,			\
-	.busy_factor		= 320,			\
+	.min_interval		= 8,			\
+	.max_interval		= 8*(min(num_online_cpus(), 32)), \
+	.busy_factor		= 64,			\
 	.imbalance_pct		= 125,			\
 	.cache_hot_time		= (10*1000000),		\
-	.cache_nice_tries	= 1,			\
+	.cache_nice_tries	= 2,			\
+	.busy_idx		= 3,			\
+	.idle_idx		= 2,			\
+	.newidle_idx		= 0, /* unused */	\
+	.wake_idx		= 1,			\
+	.forkexec_idx		= 1,			\
 	.per_cpu_gain		= 100,			\
 	.flags			= SD_LOAD_BALANCE	\
 				| SD_BALANCE_EXEC	\
-				| SD_BALANCE_NEWIDLE	\
-				| SD_WAKE_IDLE		\
+				| SD_BALANCE_FORK	\
 				| SD_WAKE_BALANCE,	\
 	.last_balance		= jiffies,		\
-	.balance_interval	= 1,			\
+	.balance_interval	= 64,			\
 	.nr_balance_failed	= 0,			\
 }
 
@@ -69,17 +99,21 @@ void build_cpu_to_node_map(void);
 	.span			= CPU_MASK_NONE,	\
 	.parent			= NULL,			\
 	.groups			= NULL,			\
-	.min_interval		= 80,			\
-	.max_interval		= 320,			\
-	.busy_factor		= 320,			\
-	.imbalance_pct		= 125,			\
+	.min_interval		= 64,			\
+	.max_interval		= 64*num_online_cpus(),	\
+	.busy_factor		= 128,			\
+	.imbalance_pct		= 133,			\
 	.cache_hot_time		= (10*1000000),		\
 	.cache_nice_tries	= 1,			\
+	.busy_idx		= 3,			\
+	.idle_idx		= 3,			\
+	.newidle_idx		= 0, /* unused */	\
+	.wake_idx		= 0, /* unused */	\
+	.forkexec_idx		= 0, /* unused */	\
 	.per_cpu_gain		= 100,			\
-	.flags			= SD_LOAD_BALANCE	\
-				| SD_BALANCE_EXEC,	\
+	.flags			= SD_LOAD_BALANCE,	\
 	.last_balance		= jiffies,		\
-	.balance_interval	= 100*(63+num_online_cpus())/64,   \
+	.balance_interval	= 64,			\
 	.nr_balance_failed	= 0,			\
 }
 
arch/ia64/kernel/domain.c renamed from arch/ia64/kernel/domain.c~sched-tune-ia64 +1 -1
diff -puN arch/ia64/kernel/domain.c~sched-tune-ia64 arch/ia64/kernel/domain.c
@@ -14,7 +14,7 @@
 #include <linux/topology.h>
 #include <linux/nodemask.h>
 
-#define SD_NODES_PER_DOMAIN 6
+#define SD_NODES_PER_DOMAIN 16
 
 #ifdef CONFIG_NUMA
 /**

_

#1 lore raw

Nick Piggin wrote on Friday, February 25, 2005 12:15 AM

include/asm-ia64/topology.h line 53

Quoted 4 lines

These are some values I just found from thin-air that should hopefully make ia64 work after the latest sched patches.

include/asm-ia64/topology.h renamed from include/asm-ia64/topology.h~sched-tune-ia64 +1 -0
+	.cache_hot_time		= (10*1000000),		\

In 2.6.9, cache_hot_time gets the value from cache_decay_tick, since when this become a constant again?

- Ken

#2 lore raw
In reply to Nick Piggin
Quoted 11 lines · Kenneth W

Nick Piggin wrote on Friday, February 25, 2005 12:15 AM

include/asm-ia64/topology.h line 53

Quoted 4 lines

These are some values I just found from thin-air that should hopefully make ia64 work after the latest sched patches.

include/asm-ia64/topology.h renamed from include/asm-ia64/topology.h~sched-tune-ia64 +1 -0
+	.cache_hot_time		= (10*1000000),		\

In 2.6.9, cache_hot_time gets the value from cache_decay_tick, since when this become a constant again?

Not sure. cache_decay_ticks isn't set to anything meaningful on ia64 anyway. Nor i386 for that matter.

Nick

#3 lore raw
In reply to Nick Piggin

Nick Piggin wrote on Friday, February 25, 2005 3:42 PM

Quoted 10 lines
Quoted 9 lines

Review

Quoted 4 lines

These are some values I just found from thin-air that should hopefully make ia64 work after the latest sched patches.

(quoted diff) +1 -0
+	.cache_hot_time		= (10*1000000),		\

In 2.6.9, cache_hot_time gets the value from cache_decay_tick, since when this become a constant again?

Quoted 2 lines

Not sure. cache_decay_ticks isn't set to anything meaningful on ia64 anyway. Nor i386 for that matter.

This parameter is known to be sensitive for one db workload. It was discussed about 4 month back. The conclusion was that there is NO one magic constant that works for every workloads on every platforms. Until a new auto tune algorithm is written/tested, it was decided that we use cache_decay_ticks as a boot time tunable for cache_hot_time.

I guess it somehow got lost soon after 2.6.9. This needs to be reinstated.

include/linux/topology.h +1 -1
@@ -109,7 +109,7 @@
 	.max_interval		= 4,			\
 	.busy_factor		= 64,			\
 	.imbalance_pct		= 125,			\
-	.cache_hot_time		= (5*1000000/2),	\
+	.cache_hot_time		= cache_decay_ticks*1000000 ? : (5*1000000/2),\
 	.cache_nice_tries	= 1,			\
 	.per_cpu_gain		= 100,			\
 	.flags			= SD_LOAD_BALANCE	\
#4 lore raw
In reply to Nick Piggin

Nick Piggin wrote on Friday, February 25, 2005 12:15 AM

include/asm-ia64/topology.h line 52

Quoted 8 lines

These are some values I just found from thin-air that should hopefully make ia64 work after the latest sched patches.

include/asm-ia64/topology.h renamed from include/asm-ia64/topology.h~sched-tune-ia64 +4 -0
+#define SD_CPU_INIT (struct sched_domain) {		\
...
+	.max_interval		= 4,			\
+	.busy_factor		= 64,			\
+	.imbalance_pct		= 125,			\

For one db workload, performance increases with the following values compare to the values proposed above.

.max_interval = 100
.busy_factor = 32
.imbalance_pct = 100

#5 lore raw
In reply to Nick Piggin
Quoted 5 lines · Kenneth W

Nick Piggin wrote on Friday, February 25, 2005 12:15 AM

Quoted 3 lines

These are some values I just found from thin-air that should hopefully make ia64 work after the latest sched patches.

Quoted 10 lines

include/asm-ia64/topology.h line 52

Quoted 5 lines
include/asm-ia64/topology.h renamed from include/asm-ia64/topology.h~sched-tune-ia64 +4 -0
+#define SD_CPU_INIT (struct sched_domain) {		\
...
+	.max_interval		= 4,			\
+	.busy_factor		= 64,			\
+	.imbalance_pct		= 125,			\

For one db workload, performance increases with the following values compare to the values proposed above.

Sorry if I wasn't clear earlier - this patch is not meant to go into 2.6. It is for Andrew to get ia64 working after my last round of scheduler patches.

There are pretty significant changes, so the optimal values for various things will have changed quite a bit.

Quoted 4 lines

.max_interval = 100
.busy_factor = 32
.imbalance_pct = 100

I can imagine max_interval could probably be increased.

Setting imbalance_pct to 100 is interesting, I wonder why this helps performance... It would be good to get schedstats output.

Nick