Skip to contentSkip to conversation

Performance regression of specjbb2005/aim7 with 2.6.29-rc1

DISCUSSION Applied Zhang, Yanmin 6 participants 10 messages Jan 13, 2009 – Jan 16, 2009

lore
Share & tools
lore

Zhang, Yanmin

author Tested-by Jayson King
#0 lore raw

Comparing with 2.6.28's results, specjbb2005 has about 7% regression with 2.6.29-rc1 on my a couple of x86_64 machines. aim7 has about 1.7% regression.

Ming did a quick bisect with aim7 and located below patch.

commit 0a582440ff546e2c6610d1acec325e91b4efd313

Author: Mike Galbraith <[email protected]>
Date:   Fri Jan 2 12:16:42 2009 +0100

sched: fix sched_slice()

Impact: fix bad-interactivity buglet

Fix sched_slice() to emit a sane result whether a task is currently enqueued or not.

After we revert the patch, aim7 regression disappeared. specjbb2005 regression becomes less than 1.5% on 8-core stokley and disappears on 16-core tigerton. I don't know what causes the last 1.5% regression.

As tbench has about 5% improvement and oltp(mysql+sysbench) has 5% improvement, we also tested to make sure such improvement isn't related to above patch. volanoMark's improvement is also not related to the patch. So it seems safe to revert it.

yanmin

#1 lore raw
Quoted 21 lines · Yanmin

Comparing with 2.6.28's results, specjbb2005 has about 7% regression with 2.6.29-rc1 on my a couple of x86_64 machines. aim7 has about 1.7% regression.

Ming did a quick bisect with aim7 and located below patch.

commit 0a582440ff546e2c6610d1acec325e91b4efd313

Author: Mike Galbraith <[email protected]>
Date:   Fri Jan 2 12:16:42 2009 +0100

sched: fix sched_slice()

Impact: fix bad-interactivity buglet

Fix sched_slice() to emit a sane result whether a task is currently enqueued or not.

Quoted 7 lines

After we revert the patch, aim7 regression disappeared. specjbb2005 regression becomes less than 1.5% on 8-core stokley and disappears on 16-core tigerton. I don't know what causes the last 1.5% regression.

As tbench has about 5% improvement and oltp(mysql+sysbench) has 5% improvement, we also tested to make sure such improvement isn't related to above patch. volanoMark's improvement is also not related to the patch. So it seems safe to revert it.

No, it's not safe to just revert. You can replace it with something else, but as long as sched_slice() is called for unqueued tasks, it must emit sane slices, otherwise you can experience a latency-hit-from-hell.

See thread: problem with "sched: revert back to per-rq vruntime"?

-Mike

#2 lore raw
Quoted 30 lines · Mike Galbraith
Quoted 21 lines · Yanmin

Comparing with 2.6.28's results, specjbb2005 has about 7% regression with 2.6.29-rc1 on my a couple of x86_64 machines. aim7 has about 1.7% regression.

Ming did a quick bisect with aim7 and located below patch.

commit 0a582440ff546e2c6610d1acec325e91b4efd313

Author: Mike Galbraith <[email protected]>
Date:   Fri Jan 2 12:16:42 2009 +0100

sched: fix sched_slice()

Impact: fix bad-interactivity buglet

Fix sched_slice() to emit a sane result whether a task is currently enqueued or not.

Quoted 7 lines

After we revert the patch, aim7 regression disappeared. specjbb2005 regression becomes less than 1.5% on 8-core stokley and disappears on 16-core tigerton. I don't know what causes the last 1.5% regression.

As tbench has about 5% improvement and oltp(mysql+sysbench) has 5% improvement, we also tested to make sure such improvement isn't related to above patch. volanoMark's improvement is also not related to the patch. So it seems safe to revert it.

Quoted 5 lines

No, it's not safe to just revert. You can replace it with something else, but as long as sched_slice() is called for unqueued tasks, it must emit sane slices, otherwise you can experience a latency-hit-from-hell.

See thread: problem with "sched: revert back to per-rq vruntime"?

Below patch fixes aim7 regression and specjbb2005 regression becomes less than 1.5% on 8-core stokley.

Jayson,
Mike's patch fixed the latency problem you reported
would you please help to test this patch to see if it still works fine now?

kernel/sched_fair.c +2 -2
index 8e1352c..617e54c 100644
@@ -429,10 +429,10 @@ static u64 sched_slice(struct cfs_rq *cfs_rq, struct sched_entity *se)
 	u64 slice = __sched_period(cfs_rq->nr_running + !se->on_rq);
 
 	for_each_sched_entity(se) {
-		struct load_weight *load = &cfs_rq->load;
+		struct load_weight *load = &cfs_rq_of(se)->load;
 
 		if (unlikely(!se->on_rq)) {
-			struct load_weight lw = cfs_rq->load;
+			struct load_weight lw = cfs_rq_of(se)->load;
 
 			update_load_add(&lw, se->load.weight);
 			load = &lw;

Thanks,
Lin Ming

Quoted 3 lines

-Mike

#3 lore raw
Quoted 52 lines · Lin Ming
Quoted 30 lines · Mike Galbraith
Quoted 21 lines · Yanmin

Comparing with 2.6.28's results, specjbb2005 has about 7% regression with 2.6.29-rc1 on my a couple of x86_64 machines. aim7 has about 1.7% regression.

Ming did a quick bisect with aim7 and located below patch.

commit 0a582440ff546e2c6610d1acec325e91b4efd313

Author: Mike Galbraith <[email protected]>
Date:   Fri Jan 2 12:16:42 2009 +0100

sched: fix sched_slice()

Impact: fix bad-interactivity buglet

Fix sched_slice() to emit a sane result whether a task is currently enqueued or not.

Quoted 7 lines

After we revert the patch, aim7 regression disappeared. specjbb2005 regression becomes less than 1.5% on 8-core stokley and disappears on 16-core tigerton. I don't know what causes the last 1.5% regression.

As tbench has about 5% improvement and oltp(mysql+sysbench) has 5% improvement, we also tested to make sure such improvement isn't related to above patch. volanoMark's improvement is also not related to the patch. So it seems safe to revert it.

Quoted 5 lines

No, it's not safe to just revert. You can replace it with something else, but as long as sched_slice() is called for unqueued tasks, it must emit sane slices, otherwise you can experience a latency-hit-from-hell.

See thread: problem with "sched: revert back to per-rq vruntime"?

Below patch fixes aim7 regression and specjbb2005 regression becomes less than 1.5% on 8-core stokley.

Jayson,
Mike's patch fixed the latency problem you reported
would you please help to test this patch to see if it still works fine now?

kernel/sched_fair.c +0 -0
index 8e1352c..617e54c 100644
@@ -429,10 +429,10 @@ static u64 sched_slice(struct cfs_rq *cfs_rq, struct sched_entity *se)
 	u64 slice = __sched_period(cfs_rq->nr_running + !se->on_rq);
 

kernel/sched_fair.c static u64 sched_slice(struct cfs_rq *cfs_rq, struct sched_entity *se) · line 438

Quoted 10 lines
kernel/sched_fair.c +2 -2
 	for_each_sched_entity(se) {
-		struct load_weight *load = &cfs_rq->load;
+		struct load_weight *load = &cfs_rq_of(se)->load;
 
 		if (unlikely(!se->on_rq)) {
-			struct load_weight lw = cfs_rq->load;
+			struct load_weight lw = cfs_rq_of(se)->load;
 
 			update_load_add(&lw, se->load.weight);
 			load = &lw;

Ugh, that's not making sense, the thing is, if !se->on_rq it doesn't yet have a sensible cfs_rq_of().

#4 lore raw
In reply to Lin Ming
Quoted 8 lines · Lin Ming
kernel/sched_fair.c +0 -0
index 8e1352c..617e54c 100644
@@ -429,10 +429,10 @@ static u64 sched_slice(struct cfs_rq *cfs_rq, struct sched_entity *se)
 	u64 slice = __sched_period(cfs_rq->nr_running + !se->on_rq);
 

kernel/sched_fair.c static u64 sched_slice(struct cfs_rq *cfs_rq, struct sched_entity *se) · line 438

Quoted 10 lines
kernel/sched_fair.c +2 -2
 	for_each_sched_entity(se) {
-		struct load_weight *load = &cfs_rq->load;
+		struct load_weight *load = &cfs_rq_of(se)->load;
 
 		if (unlikely(!se->on_rq)) {
-			struct load_weight lw = cfs_rq->load;
+			struct load_weight lw = cfs_rq_of(se)->load;
 
 			update_load_add(&lw, se->load.weight);
 			load = &lw;

I was wrong, this does look good.

Thanks

#5 lore raw
In reply to Lin Ming
Quoted 2 lines · Lin Ming

Below patch fixes aim7 regression and specjbb2005 regression becomes less than 1.5% on 8-core stokley.

Ingo, please apply the below.

Ming, would you also provide a S-o-b line?

---
Subject: sched_slice fixlet
From: Lin Ming <[email protected]>
Date: Thu Jan 15 17:10:02 CET 2009

Mike's change: 0a582440f -- sched: fix sched_slice()) Broke group scheduling by forgetting to reload cfs_rq on each loop.

---
 kernel/sched_fair.c |    5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)
kernel/sched_fair.c +4 -1
Index: linux-2.6/kernel/sched_fair.c
===================================================================
@@ -429,7 +429,10 @@ static u64 sched_slice(struct cfs_rq *cf
 	u64 slice = __sched_period(cfs_rq->nr_running + !se->on_rq);
 
 	for_each_sched_entity(se) {
-		struct load_weight *load = &cfs_rq->load;
+		struct load_weight *load;
+
+		cfs_rq = cfs_rq_of(se);
+		load = &cfs_rq->load;
 
 		if (unlikely(!se->on_rq)) {
 			struct load_weight lw = cfs_rq->load;
Applied
#6 lore raw
Quoted 6 lines · Peter Zijlstra
Quoted 2 lines · Lin Ming

Below patch fixes aim7 regression and specjbb2005 regression becomes less than 1.5% on 8-core stokley.

Ingo, please apply the below.

Applied to tip/sched/urgent, thanks!

Ingo

Tested-by
#7 lore raw
In reply to Peter Zijlstra
Quoted 31 lines · Peter Zijlstra
Quoted 2 lines · Lin Ming

Below patch fixes aim7 regression and specjbb2005 regression becomes less than 1.5% on 8-core stokley.

Ingo, please apply the below.

Ming, would you also provide a S-o-b line?

---
Subject: sched_slice fixlet
From: Lin Ming <[email protected]>
Date: Thu Jan 15 17:10:02 CET 2009

Mike's change: 0a582440f -- sched: fix sched_slice()) Broke group scheduling by forgetting to reload cfs_rq on each loop.

---
 kernel/sched_fair.c |    5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

Index: linux-2.6/kernel/sched_fair.c
===================================================================

kernel/sched_fair.c +0 -0
@@ -429,7 +429,10 @@ static u64 sched_slice(struct cfs_rq *cf
 	u64 slice = __sched_period(cfs_rq->nr_running + !se->on_rq);

kernel/sched_fair.c static u64 sched_slice(struct cfs_rq *cf · line 438

Quoted 10 lines
kernel/sched_fair.c +4 -1
 
 	for_each_sched_entity(se) {
-		struct load_weight *load = &cfs_rq->load;
+		struct load_weight *load;
+
+		cfs_rq = cfs_rq_of(se);
+		load = &cfs_rq->load;
 
 		if (unlikely(!se->on_rq)) {
 			struct load_weight lw = cfs_rq->load;

That still works for me. You may add:

#8 lore raw
Quoted 14 lines · Jayson King

kernel/sched_fair.c static u64 sched_slice(struct cfs_rq *cf · line 438

Quoted 10 lines
kernel/sched_fair.c +4 -1
-		struct load_weight *load = &cfs_rq->load;
+		struct load_weight *load;
+
+		cfs_rq = cfs_rq_of(se);
+		load = &cfs_rq->load;
  		if (unlikely(!se->on_rq)) {
 			struct load_weight lw = cfs_rq->load;

That still works for me. You may add:

Quoted 1 line

I have added it to the commit - thanks Jayson for the testing,

Ingo

#9 lore raw
In reply to Peter Zijlstra
Quoted 8 lines · Peter Zijlstra
Quoted 2 lines · Lin Ming

Below patch fixes aim7 regression and specjbb2005 regression becomes less than 1.5% on 8-core stokley.

Ingo, please apply the below.

Ming, would you also provide a S-o-b line?

Yes,

Lin Ming

Quoted 32 lines

---
Subject: sched_slice fixlet
From: Lin Ming <[email protected]>
Date: Thu Jan 15 17:10:02 CET 2009

Mike's change: 0a582440f -- sched: fix sched_slice()) Broke group scheduling by forgetting to reload cfs_rq on each loop.

---
 kernel/sched_fair.c |    5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

Index: linux-2.6/kernel/sched_fair.c
===================================================================

kernel/sched_fair.c +4 -1
@@ -429,7 +429,10 @@ static u64 sched_slice(struct cfs_rq *cf
 	u64 slice = __sched_period(cfs_rq->nr_running + !se->on_rq);
 
 	for_each_sched_entity(se) {
-		struct load_weight *load = &cfs_rq->load;
+		struct load_weight *load;
+
+		cfs_rq = cfs_rq_of(se);
+		load = &cfs_rq->load;
 
 		if (unlikely(!se->on_rq)) {
 			struct load_weight lw = cfs_rq->load;