Skip to contentSkip to conversation

Performance regression in 2.6.22-git1 (new sched code?)

DISCUSSION Martin Bligh 3 participants 4 messages Jul 10, 2007 – Jul 11, 2007

lore
Share & tools
lore
#1 lore raw
Quoted 8 lines · Martin Bligh

~ 1% on 4-way x86_64

http://test.kernel.org/perf/kernbench.elm3b6.png

~ 4% on 16-way NUMA-Q (i386)

http://test.kernel.org/perf/kernbench.moe.png

Quoted 8 lines

~ 1.5% on 4-way i386

http://test.kernel.org/perf/kernbench.elm3b132.png

There's readprofiles and stuff available from here:
http://test.kernel.org/functional/index.html
if you drill down under the tests marked "regression", eg.
http://test.kernel.org/abat/98796/002.kernbench.test/profiling/profile.text

Ok, I've done a crude comparison of a bunch of the jobs we run between 2.6.22 and 2.6.22-git1 kernbench runs, extracting the wallclock, user and system times for each. First column is 2.6.22, second is 2.6.22-git1, third is the delta. +ve bad, -ve good. Note that I have hand correlated these jobs so error is possible, but it matches Martin's figures I belive.

bl6-13

	User    85.14  88.53   3.99
	System  34.16  34.71   1.60
	Total  119.30 123.24   3.30
elm3b132
	User   2397.82 2431.95   1.42
	System 207.41 206.78  -0.30
	Total  2605.23 2638.73   1.29
elm3b6
	User   352.18 357.18   1.42
	System  37.06  36.61  -1.22
	Total  389.24 393.79   1.17
oia5-hs20
	User   645.02 653.47   1.31
	System  49.30  52.45   6.37
	Total  694.33 705.92   1.67
elm3b239
	User   878.21 950.80   8.27
	System 416.06 348.76 -16.18
	Total  1294.27 1299.56   0.41

-apw

#2 lore raw
In reply to Martin Bligh
Quoted 11 lines · Martin Bligh

~ 1% on 4-way x86_64

http://test.kernel.org/perf/kernbench.elm3b6.png

~ 4% on 16-way NUMA-Q (i386)

http://test.kernel.org/perf/kernbench.moe.png

~ 1.5% on 4-way i386

http://test.kernel.org/perf/kernbench.elm3b132.png

thx! I'll check this tomorrow, meanwhile here are a few quick ideas.

1) does reverting this patch improve performance:

commit 9c4801cebc2add1fe514bc8eb201b16372eee11a
Author: Ingo Molnar <[email protected]>
Date: Mon Jul 9 18:52:01 2007 +0200

sched: more agressive idle balancing

2) does changing CONFIG_HZ=250 to CONFIG_HZ=100 improve the numbers? CFS has some internal tuning that depends on HZ - a higher HZ is a sign that the user wants more finegrained scheduling. So for maximum server throughput, use CONFIG_HZ=100.

3) could you turn off CONFIG_SCHED_DEBUG? The cost is small but perhaps measurable.

(these 3 suggestions could be tested together - they should have cumulative effects.)

Ingo

#3 lore raw
Quoted 3 lines · Ingo Molnar
Quoted 1 line

http://test.kernel.org/perf/kernbench.elm3b132.png

thx! I'll check this tomorrow, meanwhile here are a few quick ideas.

ok, i've re-tested this, and could you please try the fix attached below, combined with a CONFIG_HZ=100 run and with SCHED_DEBUG disabled? I think a fair bit of the overhead comes from granularity being capped to 10 msecs instead of the intended 100 msecs.

Ingo

--------------------------------------------->
Subject: [patch] sched: allow larger granularity
From: Ingo Molnar <[email protected]>

allow granularity up to 100 msecs, instead of 10 msecs. (needed on larger boxes)

---
 kernel/sched.c |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/sched.c +1 -1
Index: linux/kernel/sched.c
===================================================================
@@ -4772,7 +4772,7 @@ cpumask_t nohz_cpu_mask = CPU_MASK_NONE;
 static inline void sched_init_granularity(void)
 {
 	unsigned int factor = 1 + ilog2(num_online_cpus());
-	const unsigned long gran_limit = 10000000;
+	const unsigned long gran_limit = 100000000;
 
 	sysctl_sched_granularity *= factor;
 	if (sysctl_sched_granularity > gran_limit)