Skip to contentSkip to conversation

sysbench+mysql(oltp, readonly) 30% regression with 2.6.26-rc1

DISCUSSION Zhang, Yanmin 4 participants 27 messages May 7, 2008 – May 9, 2008

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

Comparing with kernel 2.6.25, sysbench+mysql(oltp, readonly) has many regression with 2.6.26-rc1.

1) 8-core stoakley: 28%;
2) 16-core tigerton: 20%;
3) Itanium Montvale: 50%.

Bisect located below patch.

8f1bc385cfbab474db6c27b5af1e439614f3025c is first bad commit commit 8f1bc385cfbab474db6c27b5af1e439614f3025c

Author: Peter Zijlstra <[email protected]>
Date:   Sat Apr 19 19:45:00 2008 +0200

sched: fair: weight calculations

In order to level the hierarchy, we need to calculate load based on the root view. That is, each task's load is in the same unit.

After I manually reverted the patch against 2.6.26-rc1 while fixing a couple of conflictions/errors, sysbench oltp regression became less than 3% on 8-core stoakley.

-yanmin

#1 lore raw
Quoted 8 lines · Zhang, Yanmin

???Comparing with kernel 2.6.25, sysbench+mysql(oltp, readonly) has many regression with 2.6.26-rc1.

1) 8-core stoakley: 28%;
2) 16-core tigerton: 20%;
3) Itanium Montvale: 50%.

Bisect located below patch.

thanks Yanmin, i've queued up your reverter patch.

Ingo

#2 lore raw
Quoted 5 lines · Ingo Molnar
Quoted 3 lines · Zhang, Yanmin

???Comparing with kernel 2.6.25, sysbench+mysql(oltp, readonly) has many regression with 2.6.26-rc1.

Quoted 7 lines
Quoted 5 lines

1) 8-core stoakley: 28%;
2) 16-core tigerton: 20%;
3) Itanium Montvale: 50%.

Bisect located below patch.

thanks Yanmin, i've queued up your reverter patch.

Sorry. The reverting patch has a comment-out block. I need delete it if you queue the patch officially.

kernel/sched.c +6 -3
diff -Nraup linux-2.6.26-rc1/kernel/sched.c linux-2.6.26-rc1_oltp/kernel/sched.c
@@ -1429,9 +1429,6 @@ static void __resched_task(struct task_s
  */
 #define SRR(x, y) (((x) + (1UL << ((y) - 1))) >> (y))
 
-/*
- * delta *= weight / lw
- */
 static unsigned long
 calc_delta_mine(unsigned long delta_exec, unsigned long weight,
 		struct load_weight *lw)
@@ -1454,6 +1451,12 @@ calc_delta_mine(unsigned long delta_exec
 	return (unsigned long)min(tmp, (u64)(unsigned long)LONG_MAX);
 }
 
+static inline unsigned long
+calc_delta_fair(unsigned long delta_exec, struct load_weight *lw)
+{
+	return calc_delta_mine(delta_exec, NICE_0_LOAD, lw);
+}
+
 static inline void update_load_add(struct load_weight *lw, unsigned long inc)
 {
 	lw->weight += inc;
kernel/sched_fair.c +33 -67

156 lines not on this page · raw message

#3 lore raw
Quoted 4 lines · Zhang, Yanmin
Quoted 1 line

thanks Yanmin, i've queued up your reverter patch.

Sorry. The reverting patch has a comment-out block. I need delete it if you queue the patch officially.

yep, i noticed that and skipped that bit already.

Ingo

#4 lore raw
In reply to Zhang, Yanmin
Quoted 16 lines · Yanmin

Comparing with kernel 2.6.25, sysbench+mysql(oltp, readonly) has many regression with 2.6.26-rc1.

1) 8-core stoakley: 28%;
2) 16-core tigerton: 20%;
3) Itanium Montvale: 50%.

Bisect located below patch.

8f1bc385cfbab474db6c27b5af1e439614f3025c is first bad commit commit 8f1bc385cfbab474db6c27b5af1e439614f3025c

Author: Peter Zijlstra <[email protected]>
Date:   Sat Apr 19 19:45:00 2008 +0200

sched: fair: weight calculations

Quoted 8 lines

In order to level the hierarchy, we need to calculate load based on the root view. That is, each task's load is in the same unit.

After I manually reverted the patch against 2.6.26-rc1 while fixing a couple of conflictions/errors, sysbench oltp regression became less than 3% on 8-core stoakley.

Does this patch help?

---
From: Mike Galbraith <[email protected]>
Subject: sched: fix weight calculations

The conversion between virtual and real time is as follows:

dvt = rw/w * dt <=> dt = w/rw * dvt

Since we want the fair sleeper granularity to be in real time, we actually need to do:

dvt = - rw/w * l

---
 kernel/sched_fair.c |   11 ++++++++---
 1 file changed, 8 insertions(+), 3 deletions(-)
kernel/sched_fair.c +8 -3
Index: linux-2.6-2/kernel/sched_fair.c
===================================================================
@@ -662,10 +662,15 @@ place_entity(struct cfs_rq *cfs_rq, stru
 	if (!initial) {
 		/* sleeps upto a single latency don't count. */
 		if (sched_feat(NEW_FAIR_SLEEPERS)) {
+			unsigned long thresh = sysctl_sched_latency;
+
+			/*
+			 * convert the sleeper threshold into virtual time
+			 */
 			if (sched_feat(NORMALIZED_SLEEPER))
-				vruntime -= calc_delta_weight(sysctl_sched_latency, se);
-			else
-				vruntime -= sysctl_sched_latency;
+				thresh = calc_delta_fair(thresh, se);
+
+			vruntime -= thresh;
 		}
 
 		/* ensure we never gain time by being placed backwards. */
#5 lore raw
Quoted 22 lines · Peter Zijlstra
Quoted 21 lines · Yanmin

Comparing with kernel 2.6.25, sysbench+mysql(oltp, readonly) has many regression with 2.6.26-rc1.

1) 8-core stoakley: 28%;
2) 16-core tigerton: 20%;
3) Itanium Montvale: 50%.

Bisect located below patch.

8f1bc385cfbab474db6c27b5af1e439614f3025c is first bad commit commit 8f1bc385cfbab474db6c27b5af1e439614f3025c

Author: Peter Zijlstra <[email protected]>
Date:   Sat Apr 19 19:45:00 2008 +0200

sched: fair: weight calculations

In order to level the hierarchy, we need to calculate load based on the root view. That is, each task's load is in the same unit.

Quoted 5 lines
Quoted 3 lines

After I manually reverted the patch against 2.6.26-rc1 while fixing a couple of conflictions/errors, sysbench oltp regression became less than 3% on 8-core stoakley.

Does this patch help?

With the patch, oltp testing result is about 50% worse than the one of pure 2.6.26-rc1.

cpu idle with 16 threads( a parameter of sysbench):
1) 2.6.25: 1%
2) 2.6.26-rc1: 33%
3) 2.6.26-rc1+new_patch: 70%.

Quoted 44 lines

---
From: Mike Galbraith <[email protected]>
Subject: sched: fix weight calculations

The conversion between virtual and real time is as follows:

dvt = rw/w * dt <=> dt = w/rw * dvt

Since we want the fair sleeper granularity to be in real time, we actually need to do:

dvt = - rw/w * l

---
 kernel/sched_fair.c |   11 ++++++++---
 1 file changed, 8 insertions(+), 3 deletions(-)

Index: linux-2.6-2/kernel/sched_fair.c
===================================================================

kernel/sched_fair.c +8 -3
@@ -662,10 +662,15 @@ place_entity(struct cfs_rq *cfs_rq, stru
 	if (!initial) {
 		/* sleeps upto a single latency don't count. */
 		if (sched_feat(NEW_FAIR_SLEEPERS)) {
+			unsigned long thresh = sysctl_sched_latency;
+
+			/*
+			 * convert the sleeper threshold into virtual time
+			 */
 			if (sched_feat(NORMALIZED_SLEEPER))
-				vruntime -= calc_delta_weight(sysctl_sched_latency, se);
-			else
-				vruntime -= sysctl_sched_latency;
+				thresh = calc_delta_fair(thresh, se);
+
+			vruntime -= thresh;
 		}
 
 		/* ensure we never gain time by being placed backwards. */
#6 lore raw
Quoted 23 lines · Yanmin
Quoted 22 lines · Peter Zijlstra
Quoted 21 lines · Yanmin

Comparing with kernel 2.6.25, sysbench+mysql(oltp, readonly) has many regression with 2.6.26-rc1.

1) 8-core stoakley: 28%;
2) 16-core tigerton: 20%;
3) Itanium Montvale: 50%.

Bisect located below patch.

8f1bc385cfbab474db6c27b5af1e439614f3025c is first bad commit commit 8f1bc385cfbab474db6c27b5af1e439614f3025c

Author: Peter Zijlstra <[email protected]>
Date:   Sat Apr 19 19:45:00 2008 +0200

sched: fair: weight calculations

In order to level the hierarchy, we need to calculate load based on the root view. That is, each task's load is in the same unit.

Quoted 7 lines
Quoted 5 lines
Quoted 3 lines

After I manually reverted the patch against 2.6.26-rc1 while fixing a couple of conflictions/errors, sysbench oltp regression became less than 3% on 8-core stoakley.

Does this patch help?

With the patch, oltp testing result is about 50% worse than the one of pure 2.6.26-rc1.

Hm. I was doing some sysbench+postgress(oltp, ro) testing on my little Q6600 box this morning, and saw a different picture.

In attached pdf, .bkl refers to Linus' BKL patch, .weight is the weight fix, both are applied to git.today. The script I used is also attached.

-Mike

#7 lore raw
Quoted 9 lines · Mike Galbraith
Quoted 2 lines

With the patch, oltp testing result is about 50% worse than the one of pure 2.6.26-rc1.

Hm. I was doing some sysbench+postgress(oltp, ro) testing on my little Q6600 box this morning, and saw a different picture.

In attached pdf, .bkl refers to Linus' BKL patch, .weight is the weight fix, both are applied to git.today. The script I used is also attached.

nice numbers. I'm curious about the following detail related to Linus's BKL-spinlock patch: am i correct that it shows a small but systematic improvement with many clients? sysbench has not shown sensitivity to any BKL detail before.

Could you perhaps try the hack below that uses /proc/sys/kernel/panic as a flag whether the BKL should cause us to block or should be spun upon.

Can you confirm that with that patch too sysbench shows sensitivity to the value of the panic flag? [this is the surest way to measure such effects as flipping the sysctl only minimally impacts the system.]

if sysbench truly is dependent on the BKL then that's another argument in favor of Linus's patch.

Ingo

-------------------------->
Subject: BKL: spin on acquire
From: Ingo Molnar <[email protected]>
Date: Wed May 07 19:05:40 CEST 2008

---
 lib/kernel_lock.c |   13 ++++++++++---
 1 file changed, 10 insertions(+), 3 deletions(-)
lib/kernel_lock.c +10 -3
Index: linux/lib/kernel_lock.c
===================================================================
@@ -46,7 +46,10 @@ int __lockfunc __reacquire_kernel_lock(v
 	task->lock_depth = -1;
 	preempt_enable_no_resched();
 
-	down(&kernel_sem);
+	if (panic_timeout) {
+		while (down_trylock(&kernel_sem))
+			cpu_relax();
+	}
 
 	preempt_disable();
 	task->lock_depth = saved_lock_depth;
@@ -67,11 +70,15 @@ void __lockfunc lock_kernel(void)
 	struct task_struct *task = current;
 	int depth = task->lock_depth + 1;
 
-	if (likely(!depth))
+	if (likely(!depth)) {
 		/*
 		 * No recursion worries - we set up lock_depth _after_
 		 */
-		down(&kernel_sem);
+		if (panic_timeout) {
+			while (down_trylock(&kernel_sem))
+				cpu_relax();
+		}
+	}
 
 	task->lock_depth = depth;
 }
#8 lore raw

Review

Quoted 4 lines · Ingo Molnar
(quoted diff) +3 -1
-	down(&kernel_sem);
+	if (panic_timeout) {
+		while (down_trylock(&kernel_sem))
+			cpu_relax();

updated one below - this will work fine in the !panic_timeout case too ;-)

Ingo

lib/kernel_lock.c +14 -3
Index: linux/lib/kernel_lock.c
===================================================================
@@ -46,7 +46,12 @@ int __lockfunc __reacquire_kernel_lock(v
 	task->lock_depth = -1;
 	preempt_enable_no_resched();
 
-	down(&kernel_sem);
+	if (panic_timeout) {
+		while (down_trylock(&kernel_sem))
+			cpu_relax();
+	} else {
+		down(&kernel_sem);
+	}
 
 	preempt_disable();
 	task->lock_depth = saved_lock_depth;
@@ -67,11 +72,17 @@ void __lockfunc lock_kernel(void)
 	struct task_struct *task = current;
 	int depth = task->lock_depth + 1;
 
-	if (likely(!depth))
+	if (likely(!depth)) {
 		/*
 		 * No recursion worries - we set up lock_depth _after_
 		 */
-		down(&kernel_sem);
+		if (panic_timeout) {
+			while (down_trylock(&kernel_sem))
+				cpu_relax();
+		} else {
+			down(&kernel_sem);
+		}
+	}
 
 	task->lock_depth = depth;
 }
#9 lore raw
Quoted 9 lines · Ingo Molnar

lib/kernel_lock.c int __lockfunc __reacquire_kernel_lock(v · line 51

Quoted 4 lines · Ingo Molnar
lib/kernel_lock.c +3 -1
-	down(&kernel_sem);
+	if (panic_timeout) {
+		while (down_trylock(&kernel_sem))
+			cpu_relax();

updated one below - this will work fine in the !panic_timeout case too ;-)

Same result.

-Mike

#10 lore raw
In reply to Ingo Molnar
Quoted 6 lines · Ingo Molnar

Could you perhaps try the hack below that uses /proc/sys/kernel/panic as a flag whether the BKL should cause us to block or should be spun upon.

Can you confirm that with that patch too sysbench shows sensitivity to the value of the panic flag? [this is the surest way to measure such effects as flipping the sysctl only minimally impacts the system.]

Difference for average of 3 runs at 128 clients is 0.3% with this patch.. iow, nope.

-Mike

#11 lore raw
In reply to Mike Galbraith
Quoted 28 lines · Mike Galbraith
Quoted 27 lines · Yanmin
Quoted 26 lines · Peter Zijlstra
Quoted 24 lines · Yanmin

Comparing with kernel 2.6.25, sysbench+mysql(oltp, readonly) has many regression with 2.6.26-rc1.

1) 8-core stoakley: 28%;
2) 16-core tigerton: 20%;
3) Itanium Montvale: 50%.

Bisect located below patch.

8f1bc385cfbab474db6c27b5af1e439614f3025c is first bad commit commit 8f1bc385cfbab474db6c27b5af1e439614f3025c

Author: Peter Zijlstra <[email protected]>
Date:   Sat Apr 19 19:45:00 2008 +0200

sched: fair: weight calculations

In order to level the hierarchy, we need to calculate load based on the root view. That is, each task's load is in the same unit.

After I manually reverted the patch against 2.6.26-rc1 while fixing a couple of conflictions/errors, sysbench oltp regression became less than 3% on 8-core stoakley.

Quoted 6 lines
Quoted 3 lines
Quoted 1 line

Does this patch help?

With the patch, oltp testing result is about 50% worse than the one of pure 2.6.26-rc1.

Hm. I was doing some sysbench+postgress(oltp, ro) testing on my little Q6600 box this morning, and saw a different picture.

How many cpu are in the Q6600?

Quoted 3 lines

In attached pdf, .bkl refers to Linus' BKL patch, .weight is the weight fix, both are applied to git.today. The script I used is also attached.

With my 8-core stoakley (using mysql):
1) 2.6.25:
Number of threads: 6

    read/write requests:                 8025024 (66874.53 per sec.)
Number of threads: 8
    read/write requests:                 9132816 (76106.14 per sec.)
Number of threads: 10
    read/write requests:                 9244998 (77040.75 per sec.)
Number of threads: 12
    read/write requests:                 8994174 (74950.36 per sec.)
Number of threads: 14
    read/write requests:                 9051322 (75426.54 per sec.)
Number of threads: 16
    read/write requests:                 9015412 (75126.93 per sec.)

2) 2.6.26-rc1:
Number of threads: 6

    read/write requests:                 5754056 (47949.87 per sec.)
Number of threads: 8
    read/write requests:                 6528480 (54403.29 per sec.)
Number of threads: 10
    read/write requests:                 6444690 (53705.16 per sec.)
Number of threads: 12
    read/write requests:                 6544258 (54534.23 per sec.)
Number of threads: 14
    read/write requests:                 6796650 (56637.65 per sec.)
Number of threads: 16
    read/write requests:                 6718110 (55983.18 per sec.)

3) 2.6.26-rc1+weight
Number of threads: 16

	read/write requests:                 3219076 (26824.22 per sec.)

I'm not sure if more cpu could introduce more contention in this test.

-yanmin

#12 lore raw
Quoted 29 lines · Yanmin
Quoted 28 lines · Mike Galbraith
Quoted 27 lines · Yanmin
Quoted 26 lines · Peter Zijlstra
Quoted 24 lines · Yanmin

Comparing with kernel 2.6.25, sysbench+mysql(oltp, readonly) has many regression with 2.6.26-rc1.

1) 8-core stoakley: 28%;
2) 16-core tigerton: 20%;
3) Itanium Montvale: 50%.

Bisect located below patch.

8f1bc385cfbab474db6c27b5af1e439614f3025c is first bad commit commit 8f1bc385cfbab474db6c27b5af1e439614f3025c

Author: Peter Zijlstra <[email protected]>
Date:   Sat Apr 19 19:45:00 2008 +0200

sched: fair: weight calculations

In order to level the hierarchy, we need to calculate load based on the root view. That is, each task's load is in the same unit.

After I manually reverted the patch against 2.6.26-rc1 while fixing a couple of conflictions/errors, sysbench oltp regression became less than 3% on 8-core stoakley.

Quoted 7 lines
Quoted 6 lines
Quoted 3 lines
Quoted 1 line

Does this patch help?

With the patch, oltp testing result is about 50% worse than the one of pure 2.6.26-rc1.

Hm. I was doing some sysbench+postgress(oltp, ro) testing on my little Q6600 box this morning, and saw a different picture.

How many cpu are in the Q6600?

That's an Intel Quad core (Kentsfield).

Quoted 31 lines
Quoted 2 lines

In attached pdf, .bkl refers to Linus' BKL patch, .weight is the weight fix, both are applied to git.today. The script I used is also attached.

With my 8-core stoakley (using mysql):
1) 2.6.25:
Number of threads: 6

    read/write requests:                 8025024 (66874.53 per sec.)
Number of threads: 8
    read/write requests:                 9132816 (76106.14 per sec.)
Number of threads: 10
    read/write requests:                 9244998 (77040.75 per sec.)
Number of threads: 12
    read/write requests:                 8994174 (74950.36 per sec.)
Number of threads: 14
    read/write requests:                 9051322 (75426.54 per sec.)
Number of threads: 16
    read/write requests:                 9015412 (75126.93 per sec.)

2) 2.6.26-rc1:
Number of threads: 6

    read/write requests:                 5754056 (47949.87 per sec.)
Number of threads: 8
    read/write requests:                 6528480 (54403.29 per sec.)
Number of threads: 10
    read/write requests:                 6444690 (53705.16 per sec.)
Number of threads: 12
    read/write requests:                 6544258 (54534.23 per sec.)
Number of threads: 14
    read/write requests:                 6796650 (56637.65 per sec.)
Number of threads: 16
    read/write requests:                 6718110 (55983.18 per sec.)
Quoted 5 lines

3) 2.6.26-rc1+weight
Number of threads: 16

	read/write requests:                 3219076 (26824.22 per sec.)

I'm not sure if more cpu could introduce more contention in this test.

those numbers make me sad :-(

Tested-by Frans Pop
#13 lore raw
In reply to Zhang, Yanmin
Quoted 6 lines · Yanmin
Quoted 2 lines · Mike Galbraith

Hm. I was doing some sysbench+postgress(oltp, ro) testing on my little Q6600 box this morning, and saw a different picture.

How many cpu are in the Q6600?

1.

Quoted 32 lines
Quoted 3 lines

In attached pdf, .bkl refers to Linus' BKL patch, .weight is the weight fix, both are applied to git.today. The script I used is also attached.

With my 8-core stoakley (using mysql):
1) 2.6.25:
Number of threads: 6

    read/write requests:                 8025024 (66874.53 per sec.)
Number of threads: 8
    read/write requests:                 9132816 (76106.14 per sec.)
Number of threads: 10
    read/write requests:                 9244998 (77040.75 per sec.)
Number of threads: 12
    read/write requests:                 8994174 (74950.36 per sec.)
Number of threads: 14
    read/write requests:                 9051322 (75426.54 per sec.)
Number of threads: 16
    read/write requests:                 9015412 (75126.93 per sec.)

2) 2.6.26-rc1:
Number of threads: 6

    read/write requests:                 5754056 (47949.87 per sec.)
Number of threads: 8
    read/write requests:                 6528480 (54403.29 per sec.)
Number of threads: 10
    read/write requests:                 6444690 (53705.16 per sec.)
Number of threads: 12
    read/write requests:                 6544258 (54534.23 per sec.)
Number of threads: 14
    read/write requests:                 6796650 (56637.65 per sec.)
Number of threads: 16
    read/write requests:                 6718110 (55983.18 per sec.)
Quoted 5 lines

3) 2.6.26-rc1+weight
Number of threads: 16

	read/write requests:                 3219076 (26824.22 per sec.)

I'm not sure if more cpu could introduce more contention in this test.

-rc1. Do you have the fix below applied?

commit a992241de614dd2b7c97a9ba64e28c0e563f19bf

Author: Peter Zijlstra <[email protected]>
Date:   Mon May 5 23:56:17 2008 +0200

sched: fix normalized sleeper

Normalized sleeper uses calc_delta*() which requires that the rq load is already updated, so move account_entity_enqueue() before place_entity()

kernel/sched_fair.c +1 -1
index 89fa32b..1295ddc 100644
@@ -682,6 +682,7 @@ enqueue_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int wakeup)
 	 * Update run-time statistics of the 'current'.
 	 */
 	update_curr(cfs_rq);
+	account_entity_enqueue(cfs_rq, se);
 
 	if (wakeup) {
 		place_entity(cfs_rq, se, 0);
@@ -692,7 +693,6 @@ enqueue_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int wakeup)
 	check_spread(cfs_rq, se);
 	if (se != cfs_rq->curr)
 		__enqueue_entity(cfs_rq, se);
-	account_entity_enqueue(cfs_rq, se);
 }
 
 static void update_avg(u64 *avg, u64 sample)
#14 lore raw
Quoted 10 lines · Mike Galbraith
Quoted 6 lines · Yanmin
Quoted 2 lines · Mike Galbraith

Hm. I was doing some sysbench+postgress(oltp, ro) testing on my little Q6600 box this morning, and saw a different picture.

How many cpu are in the Q6600?

1.

Quoted 7 lines
Quoted 5 lines

3) 2.6.26-rc1+weight
Number of threads: 16

	read/write requests:                 3219076 (26824.22 per sec.)

I'm not sure if more cpu could introduce more contention in this test.

-rc1. Do you have the fix below applied?

No.

Quoted 36 lines

commit a992241de614dd2b7c97a9ba64e28c0e563f19bf

Author: Peter Zijlstra <[email protected]>
Date:   Mon May 5 23:56:17 2008 +0200

sched: fix normalized sleeper

Normalized sleeper uses calc_delta*() which requires that the rq load is already updated, so move account_entity_enqueue() before place_entity()

kernel/sched_fair.c +1 -1
index 89fa32b..1295ddc 100644
@@ -682,6 +682,7 @@ enqueue_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int wakeup)
 	 * Update run-time statistics of the 'current'.
 	 */
 	update_curr(cfs_rq);
+	account_entity_enqueue(cfs_rq, se);
 
 	if (wakeup) {
 		place_entity(cfs_rq, se, 0);
@@ -692,7 +693,6 @@ enqueue_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int wakeup)
 	check_spread(cfs_rq, se);
 	if (se != cfs_rq->curr)
 		__enqueue_entity(cfs_rq, se);
-	account_entity_enqueue(cfs_rq, se);
 }
 
 static void update_avg(u64 *avg, u64 sample)
#15 lore raw
In reply to Mike Galbraith
Quoted 42 lines · Mike Galbraith
Quoted 6 lines · Yanmin
Quoted 2 lines · Mike Galbraith

Hm. I was doing some sysbench+postgress(oltp, ro) testing on my little Q6600 box this morning, and saw a different picture.

How many cpu are in the Q6600?

1.

Quoted 32 lines
Quoted 3 lines

In attached pdf, .bkl refers to Linus' BKL patch, .weight is the weight fix, both are applied to git.today. The script I used is also attached.

With my 8-core stoakley (using mysql):
1) 2.6.25:
Number of threads: 6

    read/write requests:                 8025024 (66874.53 per sec.)
Number of threads: 8
    read/write requests:                 9132816 (76106.14 per sec.)
Number of threads: 10
    read/write requests:                 9244998 (77040.75 per sec.)
Number of threads: 12
    read/write requests:                 8994174 (74950.36 per sec.)
Number of threads: 14
    read/write requests:                 9051322 (75426.54 per sec.)
Number of threads: 16
    read/write requests:                 9015412 (75126.93 per sec.)

2) 2.6.26-rc1:
Number of threads: 6

    read/write requests:                 5754056 (47949.87 per sec.)
Number of threads: 8
    read/write requests:                 6528480 (54403.29 per sec.)
Number of threads: 10
    read/write requests:                 6444690 (53705.16 per sec.)
Number of threads: 12
    read/write requests:                 6544258 (54534.23 per sec.)
Number of threads: 14
    read/write requests:                 6796650 (56637.65 per sec.)
Number of threads: 16
    read/write requests:                 6718110 (55983.18 per sec.)
Quoted 7 lines
Quoted 5 lines

3) 2.6.26-rc1+weight
Number of threads: 16

	read/write requests:                 3219076 (26824.22 per sec.)

I'm not sure if more cpu could introduce more contention in this test.

-rc1. Do you have the fix below applied?

Oooh - good catch, that seems to be a post -rc1 merge.

Yes this is required.

Quoted 35 lines

commit a992241de614dd2b7c97a9ba64e28c0e563f19bf

Author: Peter Zijlstra <[email protected]>
Date:   Mon May 5 23:56:17 2008 +0200

sched: fix normalized sleeper

Normalized sleeper uses calc_delta*() which requires that the rq load is already updated, so move account_entity_enqueue() before place_entity()

kernel/sched_fair.c +1 -1
index 89fa32b..1295ddc 100644
@@ -682,6 +682,7 @@ enqueue_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int wakeup)
 	 * Update run-time statistics of the 'current'.
 	 */
 	update_curr(cfs_rq);
+	account_entity_enqueue(cfs_rq, se);
 
 	if (wakeup) {
 		place_entity(cfs_rq, se, 0);
@@ -692,7 +693,6 @@ enqueue_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int wakeup)
 	check_spread(cfs_rq, se);
 	if (se != cfs_rq->curr)
 		__enqueue_entity(cfs_rq, se);
-	account_entity_enqueue(cfs_rq, se);
 }
 
 static void update_avg(u64 *avg, u64 sample)
#16 lore raw
Quoted 47 lines · Peter Zijlstra
Quoted 46 lines · Mike Galbraith
Quoted 6 lines · Yanmin
Quoted 2 lines · Mike Galbraith

Hm. I was doing some sysbench+postgress(oltp, ro) testing on my little Q6600 box this morning, and saw a different picture.

How many cpu are in the Q6600?

1.

Quoted 36 lines
Quoted 3 lines

In attached pdf, .bkl refers to Linus' BKL patch, .weight is the weight fix, both are applied to git.today. The script I used is also attached.

With my 8-core stoakley (using mysql):
1) 2.6.25:
Number of threads: 6

    read/write requests:                 8025024 (66874.53 per sec.)
Number of threads: 8
    read/write requests:                 9132816 (76106.14 per sec.)
Number of threads: 10
    read/write requests:                 9244998 (77040.75 per sec.)
Number of threads: 12
    read/write requests:                 8994174 (74950.36 per sec.)
Number of threads: 14
    read/write requests:                 9051322 (75426.54 per sec.)
Number of threads: 16
    read/write requests:                 9015412 (75126.93 per sec.)

2) 2.6.26-rc1:
Number of threads: 6

    read/write requests:                 5754056 (47949.87 per sec.)
Number of threads: 8
    read/write requests:                 6528480 (54403.29 per sec.)
Number of threads: 10
    read/write requests:                 6444690 (53705.16 per sec.)
Number of threads: 12
    read/write requests:                 6544258 (54534.23 per sec.)
Number of threads: 14
    read/write requests:                 6796650 (56637.65 per sec.)
Number of threads: 16
    read/write requests:                 6718110 (55983.18 per sec.)

3) 2.6.26-rc1+weight
Number of threads: 16

	read/write requests:                 3219076 (26824.22 per sec.)
Quoted 7 lines
Quoted 3 lines
Quoted 1 line

I'm not sure if more cpu could introduce more contention in this test.

-rc1. Do you have the fix below applied?

Oooh - good catch, that seems to be a post -rc1 merge.

Yes this is required.

With below patch+previous_fix_weight_calc, the result is improved, but is still about 10% worse than the one of pure 2.6.26-rc1.
Number of threads: 16

    read/write requests:                 6104336 (50867.11 per sec.)
Quoted 37 lines
Quoted 36 lines

commit a992241de614dd2b7c97a9ba64e28c0e563f19bf

Author: Peter Zijlstra <[email protected]>
Date:   Mon May 5 23:56:17 2008 +0200

sched: fix normalized sleeper

Normalized sleeper uses calc_delta*() which requires that the rq load is already updated, so move account_entity_enqueue() before place_entity()

kernel/sched_fair.c +1 -1
index 89fa32b..1295ddc 100644
@@ -682,6 +682,7 @@ enqueue_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int wakeup)
 	 * Update run-time statistics of the 'current'.
 	 */
 	update_curr(cfs_rq);
+	account_entity_enqueue(cfs_rq, se);
 
 	if (wakeup) {
 		place_entity(cfs_rq, se, 0);
@@ -692,7 +693,6 @@ enqueue_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int wakeup)
 	check_spread(cfs_rq, se);
 	if (se != cfs_rq->curr)
 		__enqueue_entity(cfs_rq, se);
-	account_entity_enqueue(cfs_rq, se);
 }
 
 static void update_avg(u64 *avg, u64 sample)
#17 lore raw
Quoted 12 lines
Quoted 6 lines
Quoted 2 lines

-rc1. Do you have the fix below applied?

Oooh - good catch, that seems to be a post -rc1 merge.

Yes this is required.

With below patch+previous_fix_weight_calc, the result is improved, but is still about 10% worse than the one of pure 2.6.26-rc1.
Number of threads: 16

    read/write requests:                 6104336 (50867.11 per sec.)

So puzzling - could you try -git + fix_weight_calc, as I think that is what Mike tested and proved good on his Quad.

#18 lore raw
Quoted 6 lines · Peter Zijlstra
Quoted 5 lines
Quoted 5 lines
Quoted 1 line

-rc1. Do you have the fix below applied?

Oooh - good catch, that seems to be a post -rc1 merge.

Yes this is required.

Quoted 8 lines
Quoted 5 lines

With below patch+previous_fix_weight_calc, the result is improved, but is still about 10% worse than the one of pure 2.6.26-rc1.
Number of threads: 16

    read/write requests:                 6104336 (50867.11 per sec.)

So puzzling - could you try -git + fix_weight_calc, as I think that is what Mike tested and proved good on his Quad.

If he still has group scheduling enabled, my box agrees that there's a regression. I got mysql running, and tested with group scheduling both enabled and disabled, results attached. (for some reason, I can only test up to 64 threads with mysql though)

-Mike

#19 lore raw
Quoted 13 lines · Mike Galbraith
Quoted 12 lines · Peter Zijlstra
Quoted 11 lines
Quoted 5 lines
Quoted 1 line

-rc1. Do you have the fix below applied?

Oooh - good catch, that seems to be a post -rc1 merge.

Yes this is required.

With below patch+previous_fix_weight_calc, the result is improved, but is still about 10% worse than the one of pure 2.6.26-rc1.
Number of threads: 16

    read/write requests:                 6104336 (50867.11 per sec.)
Quoted 7 lines
Quoted 2 lines

So puzzling - could you try -git + fix_weight_calc, as I think that is what Mike tested and proved good on his Quad.

If he still has group scheduling enabled, my box agrees that there's a regression. I got mysql running, and tested with group scheduling both enabled and disabled, results attached. (for some reason, I can only test up to 64 threads with mysql though)

Hm, ok. Does the patch at hand improve or regress the group config for you?

#20 lore raw
Quoted 17 lines · Peter Zijlstra
Quoted 16 lines · Mike Galbraith
Quoted 14 lines · Peter Zijlstra
Quoted 11 lines
Quoted 5 lines
Quoted 1 line

-rc1. Do you have the fix below applied?

Oooh - good catch, that seems to be a post -rc1 merge.

Yes this is required.

With below patch+previous_fix_weight_calc, the result is improved, but is still about 10% worse than the one of pure 2.6.26-rc1.
Number of threads: 16

    read/write requests:                 6104336 (50867.11 per sec.)

So puzzling - could you try -git + fix_weight_calc, as I think that is what Mike tested and proved good on his Quad.

Quoted 7 lines
Quoted 4 lines

If he still has group scheduling enabled, my box agrees that there's a regression. I got mysql running, and tested with group scheduling both enabled and disabled, results attached. (for some reason, I can only test up to 64 threads with mysql though)

Hm, ok. Does the patch at hand improve or regress the group config for you?

I'm testing got.today.

You mean 46151122e0a2e80e5a6b2889f595e371fe2b600d?

-Mike

#21 lore raw
Quoted 23 lines · Mike Galbraith
Quoted 22 lines · Peter Zijlstra
Quoted 20 lines · Mike Galbraith
Quoted 6 lines · Peter Zijlstra
Quoted 5 lines
Quoted 5 lines
Quoted 1 line

-rc1. Do you have the fix below applied?

Oooh - good catch, that seems to be a post -rc1 merge.

Yes this is required.

Quoted 8 lines
Quoted 5 lines

With below patch+previous_fix_weight_calc, the result is improved, but is still about 10% worse than the one of pure 2.6.26-rc1.
Number of threads: 16

    read/write requests:                 6104336 (50867.11 per sec.)

So puzzling - could you try -git + fix_weight_calc, as I think that is what Mike tested and proved good on his Quad.

If he still has group scheduling enabled, my box agrees that there's a regression. I got mysql running, and tested with group scheduling both enabled and disabled, results attached. (for some reason, I can only test up to 64 threads with mysql though)

Quoted 6 lines
Quoted 2 lines

Hm, ok. Does the patch at hand improve or regress the group config for you?

I'm testing got.today.

You mean 46151122e0a2e80e5a6b2889f595e371fe2b600d?

Exactly - Yanmin says it makes it worse for him, I'm wondering if you can see a similar problem with group scheduling enabled.

#22 lore raw
Quoted 27 lines · Peter Zijlstra
Quoted 26 lines · Mike Galbraith
Quoted 24 lines · Peter Zijlstra
Quoted 13 lines · Mike Galbraith
Quoted 12 lines · Peter Zijlstra
Quoted 11 lines
Quoted 5 lines
Quoted 1 line

-rc1. Do you have the fix below applied?

Oooh - good catch, that seems to be a post -rc1 merge.

Yes this is required.

With below patch+previous_fix_weight_calc, the result is improved, but is still about 10% worse than the one of pure 2.6.26-rc1.
Number of threads: 16

    read/write requests:                 6104336 (50867.11 per sec.)
Quoted 7 lines
Quoted 2 lines

So puzzling - could you try -git + fix_weight_calc, as I think that is what Mike tested and proved good on his Quad.

If he still has group scheduling enabled, my box agrees that there's a regression. I got mysql running, and tested with group scheduling both enabled and disabled, results attached. (for some reason, I can only test up to 64 threads with mysql though)

Hm, ok. Does the patch at hand improve or regress the group config for you?

Quoted 6 lines
Quoted 3 lines

I'm testing got.today.

You mean 46151122e0a2e80e5a6b2889f595e371fe2b600d?

Exactly - Yanmin says it makes it worse for him, I'm wondering if you can see a similar problem with group scheduling enabled.

Nope, same fugly curve +-jitter.

-Mike

#23 lore raw
Quoted 5 lines · Mike Galbraith
Quoted 3 lines · Peter Zijlstra
Quoted 3 lines
Quoted 2 lines

Hm, ok. Does the patch at hand improve or regress the group config for you?

Quoted 8 lines
Quoted 6 lines
Quoted 3 lines

I'm testing got.today.

You mean 46151122e0a2e80e5a6b2889f595e371fe2b600d?

Exactly - Yanmin says it makes it worse for him, I'm wondering if you can see a similar problem with group scheduling enabled.

Nope, same fugly curve +-jitter.

Seems to be smpnice, see 2.6.26.git.grp.n- in attached.

-Mike

include/linux/sched.h +0 -1
Index: linux-2.6.26.git/include/linux/sched.h
===================================================================
@@ -766,7 +766,6 @@ struct sched_domain {
 	struct sched_domain *child;	/* bottom domain must be null terminated */
 	struct sched_group *groups;	/* the balancing groups of the domain */
 	cpumask_t span;			/* span of all CPUs in this domain */
-	int first_cpu;			/* cache of the first cpu in this domain */
 	unsigned long min_interval;	/* Minimum balance interval ms */
 	unsigned long max_interval;	/* Maximum balance interval ms */
 	unsigned int busy_factor;	/* less balancing by factor if busy */
kernel/sched.c +31 -417

589 lines not on this page · raw message

kernel/sched_fair.c +44 -80

171 lines not on this page · raw message

kernel/sched_rt.c +0 -4
Index: linux-2.6.26.git/kernel/sched_rt.c
===================================================================
@@ -513,8 +513,6 @@ static void enqueue_task_rt(struct rq *r
 	 */
 	for_each_sched_rt_entity(rt_se)
 		enqueue_rt_entity(rt_se);
-
-	inc_cpu_load(rq, p->se.load.weight);
 }
 
 static void dequeue_task_rt(struct rq *rq, struct task_struct *p, int sleep)
@@ -534,8 +532,6 @@ static void dequeue_task_rt(struct rq *r
 		if (rt_rq && rt_rq->rt_nr_running)
 			enqueue_rt_entity(rt_se);
 	}
-
-	dec_cpu_load(rq, p->se.load.weight);
 }
 
 /*
kernel/sched_debug.c +1 -1
Index: linux-2.6.26.git/kernel/sched_debug.c
===================================================================
@@ -169,7 +169,7 @@ void print_cfs_rq(struct seq_file *m, in
 			cfs_rq->nr_spread_over);
 #ifdef CONFIG_FAIR_GROUP_SCHED
 #ifdef CONFIG_SMP
-	SEQ_printf(m, "  .%-30s: %lu\n", "shares", cfs_rq->shares);
+	SEQ_printf(m, "  .%-30s: %lu\n", "shares", cfs_rq->tg->shares);
 #endif
 #endif
 }
#24 lore raw
In reply to Peter Zijlstra
Quoted 17 lines · Peter Zijlstra
Quoted 16 lines · Mike Galbraith
Quoted 14 lines · Peter Zijlstra
Quoted 11 lines
Quoted 5 lines
Quoted 1 line

-rc1. Do you have the fix below applied?

Oooh - good catch, that seems to be a post -rc1 merge.

Yes this is required.

With below patch+previous_fix_weight_calc, the result is improved, but is still about 10% worse than the one of pure 2.6.26-rc1.
Number of threads: 16

    read/write requests:                 6104336 (50867.11 per sec.)

So puzzling - could you try -git + fix_weight_calc, as I think that is what Mike tested and proved good on his Quad.

Quoted 7 lines
Quoted 4 lines

If he still has group scheduling enabled, my box agrees that there's a regression. I got mysql running, and tested with group scheduling both enabled and disabled, results attached. (for some reason, I can only test up to 64 threads with mysql though)

Hm, ok. Does the patch at hand improve or regress the group config for you?

Oh, you want me to revert it and retest group?

-Mike

#25 lore raw
Quoted 23 lines · Mike Galbraith
Quoted 22 lines · Peter Zijlstra
Quoted 20 lines · Mike Galbraith
Quoted 6 lines · Peter Zijlstra
Quoted 5 lines
Quoted 5 lines
Quoted 1 line

-rc1. Do you have the fix below applied?

Oooh - good catch, that seems to be a post -rc1 merge.

Yes this is required.

Quoted 8 lines
Quoted 5 lines

With below patch+previous_fix_weight_calc, the result is improved, but is still about 10% worse than the one of pure 2.6.26-rc1.
Number of threads: 16

    read/write requests:                 6104336 (50867.11 per sec.)

So puzzling - could you try -git + fix_weight_calc, as I think that is what Mike tested and proved good on his Quad.

If he still has group scheduling enabled, my box agrees that there's a regression. I got mysql running, and tested with group scheduling both enabled and disabled, results attached. (for some reason, I can only test up to 64 threads with mysql though)

Quoted 4 lines
Quoted 2 lines

Hm, ok. Does the patch at hand improve or regress the group config for you?

Oh, you want me to revert it and retest group?

Yes, I guess that would be what I'm asking ;-)

#26 lore raw
Quoted 24 lines · Peter Zijlstra
Quoted 23 lines · Mike Galbraith
Quoted 22 lines · Peter Zijlstra
Quoted 20 lines · Mike Galbraith
Quoted 6 lines · Peter Zijlstra
Quoted 5 lines
Quoted 5 lines
Quoted 1 line

-rc1. Do you have the fix below applied?

Oooh - good catch, that seems to be a post -rc1 merge.

Yes this is required.

Quoted 8 lines
Quoted 5 lines

With below patch+previous_fix_weight_calc, the result is improved, but is still about 10% worse than the one of pure 2.6.26-rc1.
Number of threads: 16

    read/write requests:                 6104336 (50867.11 per sec.)

So puzzling - could you try -git + fix_weight_calc, as I think that is what Mike tested and proved good on his Quad.

If he still has group scheduling enabled, my box agrees that there's a regression. I got mysql running, and tested with group scheduling both enabled and disabled, results attached. (for some reason, I can only test up to 64 threads with mysql though)

Quoted 6 lines
Quoted 4 lines
Quoted 2 lines

Hm, ok. Does the patch at hand improve or regress the group config for you?

Oh, you want me to revert it and retest group?

Yes, I guess that would be what I'm asking ;-)

(yeah yeah, ok, i went dense, results sent;)