2.6.25-rc1: volanoMark 45% regression
volanoMark has 45% regression with kernel 2.6.25-rc1 on my both 8-core stoakley and 16-core Tigerton.
I used bisect to locate below patch.
commit 58e2d4ca581167c2a079f4ee02be2f0bc52e8729
Author: Srivatsa Vaddagiri <[email protected]> Date: Fri Jan 25 21:08:00 2008 +0100
sched: group scheduling, change how cpu load is calculated
hackbench has about 30% regression on 16-core tigerton, but has about 10% improvement on 8-core stoakley.
In addition, tbench has about 6% regression on my 8-core stoakley and 25% regression on 16-core stoakley. Some other benchmarks, like netperf/aim7 also have some regression. I will verify if they are all related to the patch.
-yanmin
Quoted 16 lines · Yanmin
volanoMark has 45% regression with kernel 2.6.25-rc1 on my both 8-core stoakley and 16-core Tigerton.
I used bisect to locate below patch.
commit
58e2d4ca581167c2a079f4ee02be2f0bc52e8729Author: Srivatsa Vaddagiri <[email protected]> Date: Fri Jan 25 21:08:00 2008 +0100sched: group scheduling, change how cpu load is calculated
hackbench has about 30% regression on 16-core tigerton, but has about 10% improvement on 8-core stoakley.
Quoted 6 lines
In addition, tbench has about 6% regression on my 8-core stoakley and 25% regression on 16-core stoakley. Some other benchmarks, like netperf/aim7 also have some regression. I will verify if they are all related to the patch.
-yanmin
Hi, Yamin,
Thanks for reporting the issue? Any chance we could getthe Oprofile output for the run? The exact commandline and .config being used would also help.
Signature · 4 lines
Warm Regards, Balbir Singh Linux Technology Center IBM, ISTL
Quoted 22 lines · Balbir Singh
Quoted 21 lines · Yanmin
volanoMark has 45% regression with kernel 2.6.25-rc1 on my both 8-core stoakley and 16-core Tigerton.
I used bisect to locate below patch.
commit
58e2d4ca581167c2a079f4ee02be2f0bc52e8729Author: Srivatsa Vaddagiri <[email protected]> Date: Fri Jan 25 21:08:00 2008 +0100sched: group scheduling, change how cpu load is calculated
hackbench has about 30% regression on 16-core tigerton, but has about 10% improvement on 8-core stoakley.
In addition, tbench has about 6% regression on my 8-core stoakley and 25% regression on 16-core stoakley. Some other benchmarks, like netperf/aim7 also have some regression. I will verify if they are all related to the patch.
Quoted 6 lines
Quoted 1 line
-yanmin
Hi, Yamin,
Thanks for reporting the issue? Any chance we could getthe Oprofile output for the run? The exact commandline and .config being used would also help.
Yamin,
I would also like to know against which previous version is this
regression being compared with. Is it 2.6.24? Did you have CONFIG_FAIR_USER_SCHED enabled in both cases? It would also help to know if you see the same regression with FAIR_GROUP_SCHED turned off.
Signature · 2 lines
Regards, vatsa
Quoted 15 lines · Srivatsa Vaddagiri
Quoted 14 lines · Balbir Singh
Quoted 13 lines · Yanmin
volanoMark has 45% regression with kernel 2.6.25-rc1 on my both 8-core stoakley and 16-core Tigerton.
I used bisect to locate below patch.
commit
58e2d4ca581167c2a079f4ee02be2f0bc52e8729Author: Srivatsa Vaddagiri <[email protected]> Date: Fri Jan 25 21:08:00 2008 +0100sched: group scheduling, change how cpu load is calculated
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
hackbench has about 30% regression on 16-core tigerton, but has about 10% improvement on 8-core stoakley.
In addition, tbench has about 6% regression on my 8-core stoakley and 25% regression on 16-core stoakley.
I verified tbench regression is not caused by the same patch. I am digging tbench now.
Quoted 10 lines
Some other benchmarks, like netperf/aim7
Quoted 9 lines
Quoted 4 lines
also have some regression. I will verify if they are all related to the patch.
-yanmin
Hi, Yamin,
Thanks for reporting the issue? Any chance we could getthe Oprofile output for the run?
I got oprofile data but it didn't show clear evidence.
When doing volanoMark testing, vmstat showed the good kernel's context switch is about 1100000, but the bad kernel's context switch is 720000. Good kernel's idle is about 1%, and bad kernel's idle is about 5%.
Quoted 1 line
The exact commandline and .config being used would also help.
I used some scripts to start volanoMark.
Netperf loop UDP-RR-1/512's 10% regression on 16-core tigerton is also related to the patch. If I set CONFIG_FAIR_GROUP_SCHED=n, there is no the netperf regression. I bind the netserver process to a core and bind the client to another core in another processor.
It's hard to debug into netperf regression if it's caused by scheduler.
Quoted 4 lines
Yamin,
I would also like to know against which previous version is this
regression being compared with. Is it 2.6.24?
Yes.
Quoted 2 lines
Did you have
CONFIG_FAIR_USER_SCHEDenabled in both cases?
Yes.
CONFIG_FAIR_GROUP_SCHED=yCONFIG_FAIR_USER_SCHED=y
Quoted 2 lines
It would also help to know if you
see the same regression withFAIR_GROUP_SCHEDturned off.
No regression if CONFIG_FAIR_GROUP_SCHED=n.
-yanmin
Quoted 35 lines · Yanmin
Quoted 15 lines · Srivatsa Vaddagiri
Quoted 14 lines · Balbir Singh
Quoted 13 lines · Yanmin
volanoMark has 45% regression with kernel 2.6.25-rc1 on my both 8-core stoakley and 16-core Tigerton.
I used bisect to locate below patch.
commit
58e2d4ca581167c2a079f4ee02be2f0bc52e8729Author: Srivatsa Vaddagiri <[email protected]> Date: Fri Jan 25 21:08:00 2008 +0100sched: group scheduling, change how cpu load is calculated
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
hackbench has about 30% regression on 16-core tigerton, but has about 10% improvement on 8-core stoakley.
In addition, tbench has about 6% regression on my 8-core stoakley and 25% regression on 16-core stoakley.
I verified tbench regression is not caused by the same patch. I am digging tbench now.
Quoted 10 lines
Some other benchmarks, like netperf/aim7
Quoted 9 lines
Quoted 4 lines
also have some regression. I will verify if they are all related to the patch.
-yanmin
Hi, Yamin,
Thanks for reporting the issue? Any chance we could getthe Oprofile output for the run?
I got oprofile data but it didn't show clear evidence.
Quoted 8 lines
When doing volanoMark testing, vmstat showed the good kernel's context switch is about 1100000, but the bad kernel's context switch is 720000. Good kernel's idle is about 1%, and bad kernel's idle is about 5%.
Quoted 1 line
The exact commandline and .config being used would also help.
I used some scripts to start volanoMark.
Netperf loop UDP-RR-1/512's 10% regression
netperf loopback UDP-RR-1/512 regression is 5% on 8-core stoakley and 16-core tigerton. On tulsa machine (8 cores+hyperThread), it's 18%.
If set CONFIG_FAIR_GROUP_SCHED=n, part regression of netperf disappears. If I change dst_entry like what I said in http://marc.info/?l=linux-kernel&m=120305556317006&w=2, plus CONFIG_FAIR_GROUP_SCHED=n, the result is a little better than 2.6.24.
I have a couple of different machines and sometimes benchmarks might have different behavior. Sorry for the update.
Quoted 22 lines
on 16-core tigerton is also related to the patch.
If I setCONFIG_FAIR_GROUP_SCHED=n, there is no the netperf regression. I bind the netserver process to a core and bind the client to another core in another processor.It's hard to debug into netperf regression if it's caused by scheduler.
Quoted 4 lines
Yamin,
I would also like to know against which previous version is this
regression being compared with. Is it 2.6.24?Yes.
Quoted 2 lines
Did you have
CONFIG_FAIR_USER_SCHEDenabled in both cases?Yes.
CONFIG_FAIR_GROUP_SCHED=yCONFIG_FAIR_USER_SCHED=yQuoted 2 lines
It would also help to know if you
see the same regression withFAIR_GROUP_SCHEDturned off.No regression if
CONFIG_FAIR_GROUP_SCHED=n.
Quoted 8 lines · Yanmin
Quoted 8 lines
Quoted 3 lines
Did you have
CONFIG_FAIR_USER_SCHEDenabled in both cases?Yes.
CONFIG_FAIR_GROUP_SCHED=yCONFIG_FAIR_USER_SCHED=y
Interesting that you have CONFIG_FAIR_USER_SCHED. Do you have any other users on the system (are they running anything at the time of the benchmark)? Could you turn off CONFIG_FAIR_USER_SCHED and turn on CONFIG_FAIR_GROUP_SCHED. I am going to run some tests with 2.6.25-rc2 and Volanomark and see how that goes.
Signature · 4 lines
Warm Regards, Balbir Singh Linux Technology Center IBM, ISTL
Quoted 4 lines · Balbir Singh
Quoted 3 lines · Yanmin
Quoted 3 lines
Quoted 3 lines
Did you have
CONFIG_FAIR_USER_SCHEDenabled in both cases?
Quoted 8 lines
Quoted 5 lines
Quoted 5 lines
Yes.
CONFIG_FAIR_GROUP_SCHED=yCONFIG_FAIR_USER_SCHED=yInteresting that you have
CONFIG_FAIR_USER_SCHED. Do you have any other users on the system (are they running anything at the time of the benchmark)?
No. As for volanoMark, I start volanoMark both server and client processes while starting some system measurement tools such like vmstat/iostat/sar. All the processes run under the same user.
Quoted 2 lines
Could you
turn offCONFIG_FAIR_USER_SCHEDand turn onCONFIG_FAIR_GROUP_SCHED.
I will try.
Quoted 2 lines
I am going
to run some tests with 2.6.25-rc2 and Volanomark and see how that goes.
Pls. set /proc/sys/kernel/sched_compat_yield=1 before starting test.
-yanmin
Quoted 6 lines · Yanmin
Quoted 5 lines · Balbir Singh
Quoted 4 lines · Yanmin
Quoted 4 lines
Quoted 2 lines
Did you have
CONFIG_FAIR_USER_SCHEDenabled in both cases?Yes.
Quoted 9 lines
Quoted 5 lines
Quoted 3 lines
Quoted 3 lines
CONFIG_FAIR_GROUP_SCHED=yCONFIG_FAIR_USER_SCHED=yInteresting that you have
CONFIG_FAIR_USER_SCHED. Do you have any other users on the system (are they running anything at the time of the benchmark)?No. As for volanoMark, I start volanoMark both server and client processes while starting some system measurement tools such like vmstat/iostat/sar. All the processes run under the same user.
Ok, so the bandwidth is not split between users, which is very good. I assume the same user is root?
Quoted 8 lines
Quoted 3 lines
Could you
turn offCONFIG_FAIR_USER_SCHEDand turn onCONFIG_FAIR_GROUP_SCHED.I will try.
Quoted 2 lines
I am going
to run some tests with 2.6.25-rc2 and Volanomark and see how that goes.Pls. set /proc/sys/kernel/sched_compat_yield=1 before starting test.
Thanks for the tip, I would have definitely missed this one
Signature · 4 lines
Warm Regards, Balbir Singh Linux Technology Center IBM, ISTL
Quoted 12 lines · Balbir Singh
Quoted 11 lines · Yanmin
Quoted 10 lines · Balbir Singh
Quoted 7 lines · Yanmin
Quoted 7 lines
Quoted 2 lines
Did you have
CONFIG_FAIR_USER_SCHEDenabled in both cases?Yes.
CONFIG_FAIR_GROUP_SCHED=yCONFIG_FAIR_USER_SCHED=yInteresting that you have
CONFIG_FAIR_USER_SCHED. Do you have any other users on the system (are they running anything at the time of the benchmark)?
Quoted 7 lines
Quoted 4 lines
No. As for volanoMark, I start volanoMark both server and client processes while starting some system measurement tools such like vmstat/iostat/sar. All the processes run under the same user.
Ok, so the bandwidth is not split between users, which is very good. I assume the same user is root?
No. I use an ordinary user.
Quoted 11 lines
Quoted 9 lines
Quoted 4 lines
Could you
turn offCONFIG_FAIR_USER_SCHEDand turn onCONFIG_FAIR_GROUP_SCHED.I will try.
Quoted 2 lines
I am going
to run some tests with 2.6.25-rc2 and Volanomark and see how that goes.Pls. set /proc/sys/kernel/sched_compat_yield=1 before starting test.
Thanks for the tip, I would have definitely missed this one
I use Jrockit JVM and enables hugetlb, but I don't think hugetlb has too much help on volanoMark, especially for the regression tracking.
-yanmin