Skip to contentSkip to conversation

2.6.25-rc1: volanoMark 45% regression

DISCUSSION Zhang, Yanmin 3 participants 9 messages Feb 13, 2008 – Feb 18, 2008

lore
Share & tools
lore
#0 lore raw

volanoMark has 45% regression with kernel 2.6.25-rc1 on my both 8-core stoakley and 16-core Tigerton.

I used bisect to locate below patch.

commit 58e2d4ca581167c2a079f4ee02be2f0bc52e8729

Author: Srivatsa Vaddagiri <[email protected]>
Date:   Fri Jan 25 21:08:00 2008 +0100

sched: group scheduling, change how cpu load is calculated

hackbench has about 30% regression on 16-core tigerton, but has about 10% improvement on 8-core stoakley.

In addition, tbench has about 6% regression on my 8-core stoakley and 25% regression on 16-core stoakley. Some other benchmarks, like netperf/aim7 also have some regression. I will verify if they are all related to the patch.

-yanmin

#1 lore raw
Quoted 16 lines · Yanmin

volanoMark has 45% regression with kernel 2.6.25-rc1 on my both 8-core stoakley and 16-core Tigerton.

I used bisect to locate below patch.

commit 58e2d4ca581167c2a079f4ee02be2f0bc52e8729

Author: Srivatsa Vaddagiri <[email protected]>
Date:   Fri Jan 25 21:08:00 2008 +0100

sched: group scheduling, change how cpu load is calculated

hackbench has about 30% regression on 16-core tigerton, but has about 10% improvement on 8-core stoakley.

Quoted 6 lines

In addition, tbench has about 6% regression on my 8-core stoakley and 25% regression on 16-core stoakley. Some other benchmarks, like netperf/aim7 also have some regression. I will verify if they are all related to the patch.

-yanmin

Hi, Yamin,

Thanks for reporting the issue? Any chance we could getthe Oprofile output for the run? The exact commandline and .config being used would also help.

Signature · 4 lines
	Warm Regards,
	Balbir Singh
	Linux Technology Center
	IBM, ISTL
#2 lore raw
Quoted 22 lines · Balbir Singh
Quoted 21 lines · Yanmin

volanoMark has 45% regression with kernel 2.6.25-rc1 on my both 8-core stoakley and 16-core Tigerton.

I used bisect to locate below patch.

commit 58e2d4ca581167c2a079f4ee02be2f0bc52e8729

Author: Srivatsa Vaddagiri <[email protected]>
Date:   Fri Jan 25 21:08:00 2008 +0100

sched: group scheduling, change how cpu load is calculated

hackbench has about 30% regression on 16-core tigerton, but has about 10% improvement on 8-core stoakley.

In addition, tbench has about 6% regression on my 8-core stoakley and 25% regression on 16-core stoakley. Some other benchmarks, like netperf/aim7 also have some regression. I will verify if they are all related to the patch.

Quoted 6 lines
Quoted 1 line

-yanmin

Hi, Yamin,

Thanks for reporting the issue? Any chance we could getthe Oprofile output for the run? The exact commandline and .config being used would also help.

Yamin,
I would also like to know against which previous version is this
regression being compared with. Is it 2.6.24? Did you have CONFIG_FAIR_USER_SCHED enabled in both cases? It would also help to know if you see the same regression with FAIR_GROUP_SCHED turned off.

Signature · 2 lines
Regards,
vatsa
#3 lore raw
Quoted 15 lines · Srivatsa Vaddagiri
Quoted 14 lines · Balbir Singh
Quoted 13 lines · Yanmin

volanoMark has 45% regression with kernel 2.6.25-rc1 on my both 8-core stoakley and 16-core Tigerton.

I used bisect to locate below patch.

commit 58e2d4ca581167c2a079f4ee02be2f0bc52e8729

Author: Srivatsa Vaddagiri <[email protected]>
Date:   Fri Jan 25 21:08:00 2008 +0100

sched: group scheduling, change how cpu load is calculated

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

hackbench has about 30% regression on 16-core tigerton, but has about 10% improvement on 8-core stoakley.

In addition, tbench has about 6% regression on my 8-core stoakley and 25% regression on 16-core stoakley.

I verified tbench regression is not caused by the same patch. I am digging tbench now.

Quoted 10 lines

Some other benchmarks, like netperf/aim7

Quoted 9 lines
Quoted 4 lines

also have some regression. I will verify if they are all related to the patch.

-yanmin

Hi, Yamin,

Thanks for reporting the issue? Any chance we could getthe Oprofile output for the run?

I got oprofile data but it didn't show clear evidence.

When doing volanoMark testing, vmstat showed the good kernel's context switch is about 1100000, but the bad kernel's context switch is 720000. Good kernel's idle is about 1%, and bad kernel's idle is about 5%.

Quoted 1 line

The exact commandline and .config being used would also help.

I used some scripts to start volanoMark.

Netperf loop UDP-RR-1/512's 10% regression on 16-core tigerton is also related to the patch. If I set CONFIG_FAIR_GROUP_SCHED=n, there is no the netperf regression. I bind the netserver process to a core and bind the client to another core in another processor.

It's hard to debug into netperf regression if it's caused by scheduler.

Quoted 4 lines

Yamin,
I would also like to know against which previous version is this
regression being compared with. Is it 2.6.24?

Yes.

Quoted 2 lines

Did you have
CONFIG_FAIR_USER_SCHED enabled in both cases?

Yes.

CONFIG_FAIR_GROUP_SCHED=y
CONFIG_FAIR_USER_SCHED=y

Quoted 2 lines

It would also help to know if you
see the same regression with FAIR_GROUP_SCHED turned off.

No regression if CONFIG_FAIR_GROUP_SCHED=n.

-yanmin

#4 lore raw
Quoted 35 lines · Yanmin
Quoted 15 lines · Srivatsa Vaddagiri
Quoted 14 lines · Balbir Singh
Quoted 13 lines · Yanmin

volanoMark has 45% regression with kernel 2.6.25-rc1 on my both 8-core stoakley and 16-core Tigerton.

I used bisect to locate below patch.

commit 58e2d4ca581167c2a079f4ee02be2f0bc52e8729

Author: Srivatsa Vaddagiri <[email protected]>
Date:   Fri Jan 25 21:08:00 2008 +0100

sched: group scheduling, change how cpu load is calculated

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

hackbench has about 30% regression on 16-core tigerton, but has about 10% improvement on 8-core stoakley.

In addition, tbench has about 6% regression on my 8-core stoakley and 25% regression on 16-core stoakley.

I verified tbench regression is not caused by the same patch. I am digging tbench now.

Quoted 10 lines

Some other benchmarks, like netperf/aim7

Quoted 9 lines
Quoted 4 lines

also have some regression. I will verify if they are all related to the patch.

-yanmin

Hi, Yamin,

Thanks for reporting the issue? Any chance we could getthe Oprofile output for the run?

I got oprofile data but it didn't show clear evidence.

Quoted 8 lines

When doing volanoMark testing, vmstat showed the good kernel's context switch is about 1100000, but the bad kernel's context switch is 720000. Good kernel's idle is about 1%, and bad kernel's idle is about 5%.

Quoted 1 line

The exact commandline and .config being used would also help.

I used some scripts to start volanoMark.

Netperf loop UDP-RR-1/512's 10% regression

netperf loopback UDP-RR-1/512 regression is 5% on 8-core stoakley and 16-core tigerton. On tulsa machine (8 cores+hyperThread), it's 18%.

If set CONFIG_FAIR_GROUP_SCHED=n, part regression of netperf disappears. If I change dst_entry like what I said in http://marc.info/?l=linux-kernel&m=120305556317006&w=2, plus CONFIG_FAIR_GROUP_SCHED=n, the result is a little better than 2.6.24.

I have a couple of different machines and sometimes benchmarks might have different behavior. Sorry for the update.

Quoted 22 lines

on 16-core tigerton is also related to the patch.
If I set CONFIG_FAIR_GROUP_SCHED=n, there is no the netperf regression. I bind the netserver process to a core and bind the client to another core in another processor.

It's hard to debug into netperf regression if it's caused by scheduler.

Quoted 4 lines

Yamin,
I would also like to know against which previous version is this
regression being compared with. Is it 2.6.24?

Yes.

Quoted 2 lines

Did you have
CONFIG_FAIR_USER_SCHED enabled in both cases?

Yes.

CONFIG_FAIR_GROUP_SCHED=y
CONFIG_FAIR_USER_SCHED=y

Quoted 2 lines

It would also help to know if you
see the same regression with FAIR_GROUP_SCHED turned off.

No regression if CONFIG_FAIR_GROUP_SCHED=n.

#5 lore raw
Quoted 8 lines · Yanmin
Quoted 8 lines
Quoted 3 lines

Did you have
CONFIG_FAIR_USER_SCHED enabled in both cases?

Yes.

CONFIG_FAIR_GROUP_SCHED=y
CONFIG_FAIR_USER_SCHED=y

Interesting that you have CONFIG_FAIR_USER_SCHED. Do you have any other users on the system (are they running anything at the time of the benchmark)? Could you turn off CONFIG_FAIR_USER_SCHED and turn on CONFIG_FAIR_GROUP_SCHED. I am going to run some tests with 2.6.25-rc2 and Volanomark and see how that goes.

Signature · 4 lines
	Warm Regards,
	Balbir Singh
	Linux Technology Center
	IBM, ISTL
#6 lore raw
Quoted 4 lines · Balbir Singh
Quoted 3 lines · Yanmin
Quoted 3 lines
Quoted 3 lines

Did you have
CONFIG_FAIR_USER_SCHED enabled in both cases?

Quoted 8 lines
Quoted 5 lines
Quoted 5 lines

Yes.

CONFIG_FAIR_GROUP_SCHED=y
CONFIG_FAIR_USER_SCHED=y

Interesting that you have CONFIG_FAIR_USER_SCHED. Do you have any other users on the system (are they running anything at the time of the benchmark)?

No. As for volanoMark, I start volanoMark both server and client processes while starting some system measurement tools such like vmstat/iostat/sar. All the processes run under the same user.

Quoted 2 lines

Could you
turn off CONFIG_FAIR_USER_SCHED and turn on CONFIG_FAIR_GROUP_SCHED.

I will try.

Quoted 2 lines

I am going
to run some tests with 2.6.25-rc2 and Volanomark and see how that goes.

Pls. set /proc/sys/kernel/sched_compat_yield=1 before starting test.

-yanmin

#7 lore raw
Quoted 6 lines · Yanmin
Quoted 5 lines · Balbir Singh
Quoted 4 lines · Yanmin
Quoted 4 lines
Quoted 2 lines

Did you have
CONFIG_FAIR_USER_SCHED enabled in both cases?

Yes.

Quoted 9 lines
Quoted 5 lines
Quoted 3 lines
Quoted 3 lines

CONFIG_FAIR_GROUP_SCHED=y
CONFIG_FAIR_USER_SCHED=y

Interesting that you have CONFIG_FAIR_USER_SCHED. Do you have any other users on the system (are they running anything at the time of the benchmark)?

No. As for volanoMark, I start volanoMark both server and client processes while starting some system measurement tools such like vmstat/iostat/sar. All the processes run under the same user.

Ok, so the bandwidth is not split between users, which is very good. I assume the same user is root?

Quoted 8 lines
Quoted 3 lines

Could you
turn off CONFIG_FAIR_USER_SCHED and turn on CONFIG_FAIR_GROUP_SCHED.

I will try.

Quoted 2 lines

I am going
to run some tests with 2.6.25-rc2 and Volanomark and see how that goes.

Pls. set /proc/sys/kernel/sched_compat_yield=1 before starting test.

Thanks for the tip, I would have definitely missed this one

Signature · 4 lines
	Warm Regards,
	Balbir Singh
	Linux Technology Center
	IBM, ISTL
#8 lore raw
Quoted 12 lines · Balbir Singh
Quoted 11 lines · Yanmin
Quoted 10 lines · Balbir Singh
Quoted 7 lines · Yanmin
Quoted 7 lines
Quoted 2 lines

Did you have
CONFIG_FAIR_USER_SCHED enabled in both cases?

Yes.

CONFIG_FAIR_GROUP_SCHED=y
CONFIG_FAIR_USER_SCHED=y

Interesting that you have CONFIG_FAIR_USER_SCHED. Do you have any other users on the system (are they running anything at the time of the benchmark)?

Quoted 7 lines
Quoted 4 lines

No. As for volanoMark, I start volanoMark both server and client processes while starting some system measurement tools such like vmstat/iostat/sar. All the processes run under the same user.

Ok, so the bandwidth is not split between users, which is very good. I assume the same user is root?

No. I use an ordinary user.

Quoted 11 lines
Quoted 9 lines
Quoted 4 lines

Could you
turn off CONFIG_FAIR_USER_SCHED and turn on CONFIG_FAIR_GROUP_SCHED.

I will try.

Quoted 2 lines

I am going
to run some tests with 2.6.25-rc2 and Volanomark and see how that goes.

Pls. set /proc/sys/kernel/sched_compat_yield=1 before starting test.

Thanks for the tip, I would have definitely missed this one

I use Jrockit JVM and enables hugetlb, but I don't think hugetlb has too much help on volanoMark, especially for the regression tracking.

-yanmin