tbench regression on each kernel release from 2.6.22 -> 2.6.28
Subsystems (2)
It seems that the network stack becomes slower over time? Here is a list of tbench results with various kernel versions:
2.6.22 3207.77 mb/sec
2.6.24 3185.66
2.6.25 2848.83
2.6.26 2706.09
2.6.27(rc2) 2571.03
And linux-next is:
2.6.28(l-next) 2568.74
It shows that there is still have work to be done on linux-next. Too close to upstream in performance.
Note the KT event between 2.6.24 and 2.6.25. Why is that?
Quoted 9 lines · Christoph Lameter
It seems that the network stack becomes slower over time? Here is a list of tbench results with various kernel versions:
2.6.22 3207.77 mb/sec
2.6.24 3185.66
2.6.25 2848.83
2.6.26 2706.09
2.6.27(rc2) 2571.03
Quoted 8 lines
And linux-next is:
2.6.28(l-next) 2568.74
It shows that there is still have work to be done on linux-next. Too close to upstream in performance.
Note the KT event between 2.6.24 and 2.6.25. Why is that?
is this with SLAB or with SLUB? SLUB has been known to impact network performance...
Auke
Quoted 1 line · Auke
is this with SLAB or with SLUB? SLUB has been known to impact network performance...
The original testing config was SLAB based so it was used throughout.
David Miller
maintainerFrom: Christoph Lameter <[email protected]>
Date: Mon, 11 Aug 2008 13:36:38 -0500
Quoted 9 lines
It seems that the network stack becomes slower over time? Here is a list of tbench results with various kernel versions:
2.6.22 3207.77 mb/sec
2.6.24 3185.66
2.6.25 2848.83
2.6.26 2706.09
2.6.27(rc2) 2571.03
Quoted 8 lines
And linux-next is:
2.6.28(l-next) 2568.74
It shows that there is still have work to be done on linux-next. Too close to upstream in performance.
Note the KT event between 2.6.24 and 2.6.25. Why is that?
Isn't that when some major scheduler changes went in? I'm not blaming the scheduler, but rather I'm making the point that there are other subsystems in the kernel that the networking interacts with that influences performance at such a low level. This includes the memory allocator :-)
Quoted 5 lines · David Miller
Isn't that when some major scheduler changes went in? I'm not blaming the scheduler, but rather I'm making the point that there are other subsystems in the kernel that the networking interacts with that influences performance at such a low level. This includes the memory allocator :-)
Right this covers a significant portion of the kernel. SLAB was used since .22 was pretty early for SLUB. And around 2.6.24 we had the merges of the antifrag logic.
.25 was the point where HR timers came in. By switching off hrtimers I can get some (minor) portion of performance back. There must be more things in play though.
Maybe what we are seeing is general bloat in kernel execution paths due to the growth in complexity?
David Miller
maintainerFrom: Christoph Lameter <[email protected]>
Date: Mon, 11 Aug 2008 16:33:43 -0500
Quoted 2 lines
Maybe what we are seeing is general bloat in kernel execution paths due to the growth in complexity?
It could be, and any kind of analysis into this would be great.
I had a change that RCU destroyed sockets and this added a tiny bit of latency, so I never added it even though it would have allowed a lot of simplification of socket handling (which I though would make up for RCU's latency, but it didn't).
Quoted 7 lines · David Miller
From: Christoph Lameter <[email protected]>
Date: Mon, 11 Aug 2008 16:33:43 -0500Quoted 2 lines
Maybe what we are seeing is general bloat in kernel execution paths due to the growth in complexity?
It could be, and any kind of analysis into this would be great.
perhaps Rick Jones who maintains netperf could enlighten us on some historic numbers? he usually seems to be happy to prop up new netperf numbers :)
Auke
Quoted 6 lines · Auke
Quoted 4 lines · David Miller
From: Christoph Lameter <[email protected]>
Date: Mon, 11 Aug 2008 16:33:43 -0500
Quoted 8 lines
Quoted 4 lines
Quoted 2 lines
Maybe what we are seeing is general bloat in kernel execution paths due to the growth in complexity?
It could be, and any kind of analysis into this would be great.
perhaps Rick Jones who maintains netperf could enlighten us on some historic numbers? he usually seems to be happy to prop up new netperf numbers :)
While this is an excellent opening to talk about how netperf top-of-trunk can now emit keyword=value results easier (ostensibly) to put into a database then the regular or even CSV output formats, I cannot fully exploit it by pointing at a database of results :(
rick jones
Quoted 2 lines · Christoph Lameter
Maybe what we are seeing is general bloat in kernel execution paths due to the growth in complexity?
Wouldn't surprise me. Have you considered doing profiles?
e.g. just oprofiling the benchmark on the different kernels and see if there's some obvious difference in the CPU consumers?
-Andi
Quoted 6 lines · Andi Kleen
Quoted 2 lines · Christoph Lameter
Maybe what we are seeing is general bloat in kernel execution paths due to the growth in complexity?
Wouldn't surprise me. Have you considered doing profiles?
If I get the time I will try to do that.
Another way to understand why we are accepting the regressions here may be that we give more consideration to real time issues and deterministic performance these days. Hardware speed gains compensate for the additional bloat? (I ran the old kernels on cutting edge hardware after all).
Quoted 19 lines · David Miller
From: Christoph Lameter <[email protected]>
Date: Mon, 11 Aug 2008 13:36:38 -0500Quoted 16 lines
It seems that the network stack becomes slower over time? Here is a list of tbench results with various kernel versions:
2.6.22 3207.77 mb/sec
2.6.24 3185.66
2.6.25 2848.83
2.6.26 2706.09
2.6.27(rc2) 2571.03And linux-next is:
2.6.28(l-next) 2568.74
It shows that there is still have work to be done on linux-next. Too close to upstream in performance.
Quoted 6 lines
Quoted 1 line
Note the KT event between 2.6.24 and 2.6.25. Why is that?
Isn't that when some major scheduler changes went in? I'm not blaming the scheduler, but rather I'm making the point that there are other subsystems in the kernel that the networking interacts with that influences performance at such a low level.
...IIRC, somebody in the past did even bisect his (probably netperf) 2.6.24-25 regression to some scheduler change (obviously it might or might not be related to this case of yours)...
Signature · 1 line
i.
Quoted 23 lines · Ilpo Järvinen
Quoted 21 lines · David Miller
From: Christoph Lameter <[email protected]>
Date: Mon, 11 Aug 2008 13:36:38 -0500Quoted 17 lines
It seems that the network stack becomes slower over time? Here is a list of tbench results with various kernel versions:
2.6.22 3207.77 mb/sec
2.6.24 3185.66
2.6.25 2848.83
2.6.26 2706.09
2.6.27(rc2) 2571.03And linux-next is:
2.6.28(l-next) 2568.74
It shows that there is still have work to be done on linux-next. Too close to upstream in performance.
Note the KT event between 2.6.24 and 2.6.25. Why is that?
Quoted 8 lines
Quoted 4 lines
Isn't that when some major scheduler changes went in? I'm not blaming the scheduler, but rather I'm making the point that there are other subsystems in the kernel that the networking interacts with that influences performance at such a low level.
...IIRC, somebody in the past did even bisect his (probably netperf) 2.6.24-25 regression to some scheduler change (obviously it might or might not be related to this case of yours)...
I did find much regression with netperf TCP-RR-1/UDP-RR-1/UDP-RR-512. I start 1 serve and 1 client while binding them to a different logical processor in different physical cpu.
Comparing with 2.6.22, the regression of TCP-RR-1 on 16-core tigerton is:
2.6.23 6%
2.6.24 6%
2.6.25 9.7%
2.6.26 14.5%
2.6.27-rc1 22%
Other regressions on other machines are similar.
yanmin
Quoted 37 lines · Yanmin
Quoted 23 lines · Ilpo Järvinen
Quoted 21 lines · David Miller
From: Christoph Lameter <[email protected]>
Date: Mon, 11 Aug 2008 13:36:38 -0500Quoted 17 lines
It seems that the network stack becomes slower over time? Here is a list of tbench results with various kernel versions:
2.6.22 3207.77 mb/sec
2.6.24 3185.66
2.6.25 2848.83
2.6.26 2706.09
2.6.27(rc2) 2571.03And linux-next is:
2.6.28(l-next) 2568.74
It shows that there is still have work to be done on linux-next. Too close to upstream in performance.
Note the KT event between 2.6.24 and 2.6.25. Why is that?
Quoted 8 lines
Quoted 4 lines
Isn't that when some major scheduler changes went in? I'm not blaming the scheduler, but rather I'm making the point that there are other subsystems in the kernel that the networking interacts with that influences performance at such a low level.
...IIRC, somebody in the past did even bisect his (probably netperf) 2.6.24-25 regression to some scheduler change (obviously it might or might not be related to this case of yours)...
I did find much regression with netperf TCP-RR-1/UDP-RR-1/UDP-RR-512. I start 1 serve and 1 client while binding them to a different logical processor in different physical cpu.
Quoted 8 lines
Comparing with 2.6.22, the regression of TCP-RR-1 on 16-core tigerton is:
2.6.23 6%
2.6.24 6%
2.6.25 9.7%
2.6.26 14.5%
2.6.27-rc1 22%Other regressions on other machines are similar.
I btw reorganized tcp_sock for 2.6.26, it shouldn't cause this but it's not always obvious what even a small change in field ordering does for performance (it's b79eeeb9e48457579cb742cd02e162fcd673c4a3 in case you want to check that).
Also, there was this 83f36f3f35f4f83fa346bfff58a5deabc78370e5 fix to current -rcs but I guess it might not be that significant in your case (but I don't know well enough :-)).
Signature · 1 line
i.
Quoted 48 lines · Ilpo Järvinen
Quoted 37 lines · Yanmin
Quoted 23 lines · Ilpo Järvinen
Quoted 21 lines · David Miller
From: Christoph Lameter <[email protected]>
Date: Mon, 11 Aug 2008 13:36:38 -0500Quoted 17 lines
It seems that the network stack becomes slower over time? Here is a list of tbench results with various kernel versions:
2.6.22 3207.77 mb/sec
2.6.24 3185.66
2.6.25 2848.83
2.6.26 2706.09
2.6.27(rc2) 2571.03And linux-next is:
2.6.28(l-next) 2568.74
It shows that there is still have work to be done on linux-next. Too close to upstream in performance.
Note the KT event between 2.6.24 and 2.6.25. Why is that?
Quoted 8 lines
Quoted 4 lines
Isn't that when some major scheduler changes went in? I'm not blaming the scheduler, but rather I'm making the point that there are other subsystems in the kernel that the networking interacts with that influences performance at such a low level.
...IIRC, somebody in the past did even bisect his (probably netperf) 2.6.24-25 regression to some scheduler change (obviously it might or might not be related to this case of yours)...
I did find much regression with netperf TCP-RR-1/UDP-RR-1/UDP-RR-512. I start 1 serve and 1 client while binding them to a different logical processor in different physical cpu.
Quoted 8 lines
Comparing with 2.6.22, the regression of TCP-RR-1 on 16-core tigerton is:
2.6.23 6%
2.6.24 6%
2.6.25 9.7%
2.6.26 14.5%
2.6.27-rc1 22%Other regressions on other machines are similar.
Quoted 8 lines
I btw reorganized
tcp_sockfor 2.6.26, it shouldn't cause this but it's not always obvious what even a small change in field ordering does for performance (it'sb79eeeb9e48457579cb742cd02e162fcd673c4a3in case you want to check that).Also, there was this
83f36f3f35f4f83fa346bfff58a5deabc78370e5fix to current -rcs but I guess it might not be that significant in your case (but I don't know well enough :-)).
I reverted the patch against 2.6.27-rc1 and did a quick testing with netperf TCP-RR-1 and didn't find improvement. So your patch is good.
Mostly, I suspect process scheduler causes the regression. It seems when there are only 1 or 2 tasks running on the cpu, the performance isn't good. My netperf testing is just one example.
Quoted 1 line · Yanmin
Other regressions on other machines are similar.
There are AIM7 regressions that are similar to tbench.
2.6.22 28436
2.6.26 23064
Quoted 8 lines · Christoph Lameter
Quoted 1 line · Yanmin
Other regressions on other machines are similar.
There are AIM7 regressions that are similar to tbench.
2.6.22 28436
2.6.26 23064
Just a shot in the dark -- is this with Group Scheduling on or off? Off is prefered for benchmarks.
Quoted 4 lines · Ray Lee
Quoted 2 lines · Christoph Lameter
Zhang, Yanmin wrote:
Quoted 8 lines
Quoted 5 lines
Quoted 1 line
Other regressions on other machines are similar.
There are AIM7 regressions that are similar to tbench.
2.6.22 28436
2.6.26 23064Just a shot in the dark -- is this with Group Scheduling on or off? Off is prefered for benchmarks.
Off
Quoted 8 lines · Christoph Lameter
Quoted 1 line · Yanmin
Other regressions on other machines are similar.
There are AIM7 regressions that are similar to tbench.
2.6.22 28436
2.6.26 23064
Mostly, AIM7 has about 4~5% regression on my machines. As AIM7 result is stable, so 4% is big.
Quoted 8 lines · Christoph Lameter
It seems that the network stack becomes slower over time? Here is a list of tbench results with various kernel versions:
2.6.22 3207.77 mb/sec
2.6.24 3185.66
2.6.25 2848.83
2.6.26 2706.09
2.6.27(rc2) 2571.03
What's the hardware configuration? Is it dual-core?
I also track tbench performance with lastest kernels on a couple of quad-core machines, and didn't find such regression while the results did have fluctuation.
What's the commandline you is using to start tbench? I start tbench with CPU_NUM*2.
BTW, I enabled CONFIG_SLUB since 2.6.22.
Quoted 9 lines
And linux-next is:
2.6.28(l-next) 2568.74
It shows that there is still have work to be done on linux-next. Too close to upstream in performance.
Note the KT event between 2.6.24 and 2.6.25. Why is that?