Skip to contentSkip to conversation

hackbench regression since 2.6.25-rc

DISCUSSION Zhang, Yanmin 5 participants 31 messages Mar 13, 2008 – Mar 18, 2008

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

Comparing with 2.6.24, on my 16-core tigerton, hackbench process mode has about 40% regression with 2.6.25-rc1, and more than 20% regression with kernel 2.6.25-rc4, because rc4 includes the reverting patch of scheduler load balance.

Command to start it.
#hackbench 100 process 2000
I ran it for 3 times and sum the values.

I tried to investiagte it by bisect.
Kernel up to tag 0f4dafc0563c6c49e17fe14b3f5f356e4c4b8806 has the 20% regression. Kernel up to tag 6e90aa972dda8ef86155eefcdbdc8d34165b9f39 hasn't regression.

Any bisect between above 2 tags cause kernel hang. I tried to checkout to a point between these 2 tags for many times manually and kernel always paniced.

All patches between the 2 tags are on kobject restructure. I guess such restructure creates more cache miss on the 16-core tigerton.

Any idea?

-yanmin

maintainer
#1 lore raw
Quoted 12 lines · Yanmin

Comparing with 2.6.24, on my 16-core tigerton, hackbench process mode has about 40% regression with 2.6.25-rc1, and more than 20% regression with kernel 2.6.25-rc4, because rc4 includes the reverting patch of scheduler load balance.

Command to start it.
#hackbench 100 process 2000
I ran it for 3 times and sum the values.

I tried to investiagte it by bisect.
Kernel up to tag 0f4dafc0563c6c49e17fe14b3f5f356e4c4b8806 has the 20% regression. Kernel up to tag 6e90aa972dda8ef86155eefcdbdc8d34165b9f39 hasn't regression.

Quoted 6 lines

Any bisect between above 2 tags cause kernel hang. I tried to checkout to a point between these 2 tags for many times manually and kernel always paniced.

All patches between the 2 tags are on kobject restructure. I guess such restructure creates more cache miss on the 16-core tigerton.

That's pretty surprising - hackbench spends most of its time in userspace and zeroing out anonymous pages. It shouldn't be fiddling with kobjects much at all.

Some kernel profiling might be needed here..

#2 lore raw
Quoted 17 lines · Andrew Morton
Quoted 15 lines · Yanmin

Comparing with 2.6.24, on my 16-core tigerton, hackbench process mode has about 40% regression with 2.6.25-rc1, and more than 20% regression with kernel 2.6.25-rc4, because rc4 includes the reverting patch of scheduler load balance.

Command to start it.
#hackbench 100 process 2000
I ran it for 3 times and sum the values.

I tried to investiagte it by bisect.
Kernel up to tag 0f4dafc0563c6c49e17fe14b3f5f356e4c4b8806 has the 20% regression. Kernel up to tag 6e90aa972dda8ef86155eefcdbdc8d34165b9f39 hasn't regression.

Any bisect between above 2 tags cause kernel hang. I tried to checkout to a point between these 2 tags for many times manually and kernel always paniced.

Quoted 6 lines
Quoted 3 lines

All patches between the 2 tags are on kobject restructure. I guess such restructure creates more cache miss on the 16-core tigerton.

That's pretty surprising - hackbench spends most of its time in userspace and zeroing out anonymous pages.

No. vmstat showed hackbench spends almost 100% in sys.

Quoted 4 lines

It shouldn't be fiddling with kobjects
much at all.

Some kernel profiling might be needed here..

Thanks for your kind reminder. I don't know why I forgot it.

2.6.24 oprofile data:
CPU: Core 2, speed 1602 MHz (estimated)
Counted CPU_CLK_UNHALTED events (Clock cycles when not halted) with a unit mask of 0x00 (Unhalted core cycles) count 100000

samples  %        image name               app name                 symbol name
40200494 43.3899  linux-2.6.24             linux-2.6.24             __slab_alloc
35338431 38.1421  linux-2.6.24             linux-2.6.24             add_partial_tail
2993156   3.2306  linux-2.6.24             linux-2.6.24             __slab_free
1365806   1.4742  linux-2.6.24             linux-2.6.24             sock_alloc_send_skb
1253820   1.3533  linux-2.6.24             linux-2.6.24             copy_user_generic_string
1141442   1.2320  linux-2.6.24             linux-2.6.24             unix_stream_recvmsg
846836    0.9140  linux-2.6.24             linux-2.6.24             unix_stream_sendmsg
777561    0.8393  linux-2.6.24             linux-2.6.24             kmem_cache_alloc
587127    0.6337  linux-2.6.24             linux-2.6.24             sock_def_readable

2.6.25-rc4 oprofile data:
CPU: Core 2, speed 1602 MHz (estimated)
Counted CPU_CLK_UNHALTED events (Clock cycles when not halted) with a unit mask of 0x00 (Unhalted core cycles) count 100000

samples  %        image name               app name                 symbol name
46746994 43.3801  linux-2.6.25-rc4         linux-2.6.25-rc4         __slab_alloc
45986635 42.6745  linux-2.6.25-rc4         linux-2.6.25-rc4         add_partial
2577578   2.3919  linux-2.6.25-rc4         linux-2.6.25-rc4         __slab_free
1301644   1.2079  linux-2.6.25-rc4         linux-2.6.25-rc4         sock_alloc_send_skb
1185888   1.1005  linux-2.6.25-rc4         linux-2.6.25-rc4         copy_user_generic_string
969847    0.9000  linux-2.6.25-rc4         linux-2.6.25-rc4         unix_stream_recvmsg
806665    0.7486  linux-2.6.25-rc4         linux-2.6.25-rc4         kmem_cache_alloc
731059    0.6784  linux-2.6.25-rc4         linux-2.6.25-rc4         unix_stream_sendmsg

-yanmin

maintainer
#3 lore raw
Quoted 18 lines · Yanmin
Quoted 17 lines · Andrew Morton
Quoted 15 lines · Yanmin

Comparing with 2.6.24, on my 16-core tigerton, hackbench process mode has about 40% regression with 2.6.25-rc1, and more than 20% regression with kernel 2.6.25-rc4, because rc4 includes the reverting patch of scheduler load balance.

Command to start it.
#hackbench 100 process 2000
I ran it for 3 times and sum the values.

I tried to investiagte it by bisect.
Kernel up to tag 0f4dafc0563c6c49e17fe14b3f5f356e4c4b8806 has the 20% regression. Kernel up to tag 6e90aa972dda8ef86155eefcdbdc8d34165b9f39 hasn't regression.

Any bisect between above 2 tags cause kernel hang. I tried to checkout to a point between these 2 tags for many times manually and kernel always paniced.

Quoted 7 lines
Quoted 6 lines
Quoted 3 lines

All patches between the 2 tags are on kobject restructure. I guess such restructure creates more cache miss on the 16-core tigerton.

That's pretty surprising - hackbench spends most of its time in userspace and zeroing out anonymous pages.

No. vmstat showed hackbench spends almost 100% in sys.

ah, I got confused about which test that is.

Quoted 26 lines
Quoted 4 lines

It shouldn't be fiddling with kobjects
much at all.

Some kernel profiling might be needed here..

Thanks for your kind reminder. I don't know why I forgot it.

2.6.24 oprofile data:
CPU: Core 2, speed 1602 MHz (estimated)
Counted CPU_CLK_UNHALTED events (Clock cycles when not halted) with a unit mask of 0x00 (Unhalted core cycles) count 100000

samples  %        image name               app name                 symbol name
40200494 43.3899  linux-2.6.24             linux-2.6.24             __slab_alloc
35338431 38.1421  linux-2.6.24             linux-2.6.24             add_partial_tail
2993156   3.2306  linux-2.6.24             linux-2.6.24             __slab_free
1365806   1.4742  linux-2.6.24             linux-2.6.24             sock_alloc_send_skb
1253820   1.3533  linux-2.6.24             linux-2.6.24             copy_user_generic_string
1141442   1.2320  linux-2.6.24             linux-2.6.24             unix_stream_recvmsg
846836    0.9140  linux-2.6.24             linux-2.6.24             unix_stream_sendmsg
777561    0.8393  linux-2.6.24             linux-2.6.24             kmem_cache_alloc
587127    0.6337  linux-2.6.24             linux-2.6.24             sock_def_readable

2.6.25-rc4 oprofile data:
CPU: Core 2, speed 1602 MHz (estimated)
Counted CPU_CLK_UNHALTED events (Clock cycles when not halted) with a unit mask of 0x00 (Unhalted core cycles) count 100000

Quoted 10 lines
samples  %        image name               app name                 symbol name
46746994 43.3801  linux-2.6.25-rc4         linux-2.6.25-rc4         __slab_alloc
45986635 42.6745  linux-2.6.25-rc4         linux-2.6.25-rc4         add_partial
2577578   2.3919  linux-2.6.25-rc4         linux-2.6.25-rc4         __slab_free
1301644   1.2079  linux-2.6.25-rc4         linux-2.6.25-rc4         sock_alloc_send_skb
1185888   1.1005  linux-2.6.25-rc4         linux-2.6.25-rc4         copy_user_generic_string
969847    0.9000  linux-2.6.25-rc4         linux-2.6.25-rc4         unix_stream_recvmsg
806665    0.7486  linux-2.6.25-rc4         linux-2.6.25-rc4         kmem_cache_alloc
731059    0.6784  linux-2.6.25-rc4         linux-2.6.25-rc4         unix_stream_sendmsg

So slub got a litle slower?

(Is slab any better?)

Still, I don't think there are any kobject operations in these codepaths are there? Maybe some related to the network device, but I doubt it - networking tends to go it alone on those things, mainly for performance reasons.

#5 lore raw
Quoted 4 lines · Christoph Lameter

Could you recompile the kernel with slub performance statistics and post the output of

slabinfo -AD

Before testing with kernel 2.6.25-rc5:
Name Objects Alloc Free %Fast

vm_area_struct            2795   135185   132587  93  29
:0004096                    25   119045   119043  99  98
:0000064                 12257   119671   107742  98  50
:0000192                  3312    78563    75370  92  21
:0000128                  4648    48143    43738  97  53
dentry                   15217    46675    31527  95  72
:0000080                 12784    33674    21206  99  97
:0000016                  4367    25871    23705  99  78
:0000096                  3001    22591    20084  99  92
buffer_head               5536    18147    12884  97  42
anon_vma                  1729    14948    14130  99  73

After testing:
Name Objects Alloc Free %Fast

:0000192                  3428 80093958 80090708  92   8
:0000512                   374 80016030 80015715  68   7
vm_area_struct            2875   224524   221868  94  20
:0000064                 12408   134273   122227  98  47
:0004096                    24   127397   127395  99  98
:0000128                  4596    57837    53432  97  48
dentry                   15659    51402    35824  95  64
:0000016                  4584    29327    27161  99  76
:0000080                 12784    33674    21206  99  97
:0000096                  2998    26264    23757  99  93

So block 192 and 512's and very active and their fast free percentage is low.

-yanmin

#6 lore raw
Quoted 34 lines · Yanmin
Quoted 4 lines · Christoph Lameter

Could you recompile the kernel with slub performance statistics and post the output of

slabinfo -AD

Before testing with kernel 2.6.25-rc5:
Name Objects Alloc Free %Fast

vm_area_struct            2795   135185   132587  93  29
:0004096                    25   119045   119043  99  98
:0000064                 12257   119671   107742  98  50
:0000192                  3312    78563    75370  92  21
:0000128                  4648    48143    43738  97  53
dentry                   15217    46675    31527  95  72
:0000080                 12784    33674    21206  99  97
:0000016                  4367    25871    23705  99  78
:0000096                  3001    22591    20084  99  92
buffer_head               5536    18147    12884  97  42
anon_vma                  1729    14948    14130  99  73

After testing:
Name Objects Alloc Free %Fast

:0000192                  3428 80093958 80090708  92   8
:0000512                   374 80016030 80015715  68   7
vm_area_struct            2875   224524   221868  94  20
:0000064                 12408   134273   122227  98  47
:0004096                    24   127397   127395  99  98
:0000128                  4596    57837    53432  97  48
dentry                   15659    51402    35824  95  64
:0000016                  4584    29327    27161  99  76
:0000080                 12784    33674    21206  99  97
:0000096                  2998    26264    23757  99  93
Quoted 1 line

So block 192 and 512's and very active and their fast free percentage is low.

On my 8-core stoakley, there is no such regression. Below data is after testing.

[root@lkp-st02-x8664 ~]# slabinfo -AD
Name Objects Alloc Free %Fast

:0000192                  3170 80055388 80052280  92   1 
:0000512                   316 80012750 80012466  69   1 
vm_area_struct            2642   194700   192193  94  16 
:0000064                  3846    74468    70820  97  53 
:0004096                    15    69014    69012  98  97 
:0000128                  1447    32920    31541  91   8 
dentry                   13485    33060    19652  92  42 
:0000080                 10639    23377    12953  98  98 
:0000096                  1662    16496    15036  99  94 
:0000832                   232    14422    14203  85  10 
:0000016                  2733    15102    13372  99  14 

So the block 192 and 512's fast free percentage is even smaller than the ones on tigerton.

Oprofile data on stoakley:

CPU: Core 2, speed 2660 MHz (estimated)
Counted CPU_CLK_UNHALTED events (Clock cycles when not halted) with a unit mask of 0x00 (Unhalted core cycles) count 100000

samples  %        app name                 symbol name
2897265  25.7603  linux-2.6.25-rc5         __slab_alloc
2689900  23.9166  linux-2.6.25-rc5         add_partial
629355    5.5957  linux-2.6.25-rc5         copy_user_generic_string
552309    4.9107  linux-2.6.25-rc5         __slab_free
514792    4.5771  linux-2.6.25-rc5         sock_alloc_send_skb
500879    4.4534  linux-2.6.25-rc5         unix_stream_recvmsg
274798    2.4433  linux-2.6.25-rc5         __kmalloc_track_caller
230283    2.0475  linux-2.6.25-rc5         kfree
222286    1.9764  linux-2.6.25-rc5         unix_stream_sendmsg
217413    1.9331  linux-2.6.25-rc5         memset_c
211589    1.8813  linux-2.6.25-rc5         kmem_cache_alloc
151500    1.3470  linux-2.6.25-rc5         system_call
132262    1.1760  linux-2.6.25-rc5         sock_def_readable
123130    1.0948  linux-2.6.25-rc5         kmem_cache_free
109518    0.9738  linux-2.6.25-rc5         sock_wfree

yanmin

#7 lore raw
Quoted 58 lines · Yanmin
Quoted 34 lines · Yanmin
Quoted 4 lines · Christoph Lameter

Could you recompile the kernel with slub performance statistics and post the output of

slabinfo -AD

Before testing with kernel 2.6.25-rc5:
Name Objects Alloc Free %Fast

vm_area_struct            2795   135185   132587  93  29
:0004096                    25   119045   119043  99  98
:0000064                 12257   119671   107742  98  50
:0000192                  3312    78563    75370  92  21
:0000128                  4648    48143    43738  97  53
dentry                   15217    46675    31527  95  72
:0000080                 12784    33674    21206  99  97
:0000016                  4367    25871    23705  99  78
:0000096                  3001    22591    20084  99  92
buffer_head               5536    18147    12884  97  42
anon_vma                  1729    14948    14130  99  73

After testing:
Name Objects Alloc Free %Fast

:0000192                  3428 80093958 80090708  92   8
:0000512                   374 80016030 80015715  68   7
vm_area_struct            2875   224524   221868  94  20
:0000064                 12408   134273   122227  98  47
:0004096                    24   127397   127395  99  98
:0000128                  4596    57837    53432  97  48
dentry                   15659    51402    35824  95  64
:0000016                  4584    29327    27161  99  76
:0000080                 12784    33674    21206  99  97
:0000096                  2998    26264    23757  99  93
Quoted 1 line

So block 192 and 512's and very active and their fast free percentage is low.

On my 8-core stoakley, there is no such regression. Below data is after testing.

[root@lkp-st02-x8664 ~]# slabinfo -AD
Name Objects Alloc Free %Fast

:0000192                  3170 80055388 80052280  92   1 
:0000512                   316 80012750 80012466  69   1 
vm_area_struct            2642   194700   192193  94  16 
:0000064                  3846    74468    70820  97  53 
:0004096                    15    69014    69012  98  97 
:0000128                  1447    32920    31541  91   8 
dentry                   13485    33060    19652  92  42 
:0000080                 10639    23377    12953  98  98 
:0000096                  1662    16496    15036  99  94 
:0000832                   232    14422    14203  85  10 
:0000016                  2733    15102    13372  99  14 

So the block 192 and 512's fast free percentage is even smaller than the ones on tigerton.

Oprofile data on stoakley:

CPU: Core 2, speed 2660 MHz (estimated)
Counted CPU_CLK_UNHALTED events (Clock cycles when not halted) with a unit mask of 0x00 (Unhalted core cycles) count 100000

Quoted 16 lines
samples  %        app name                 symbol name
2897265  25.7603  linux-2.6.25-rc5         __slab_alloc
2689900  23.9166  linux-2.6.25-rc5         add_partial
629355    5.5957  linux-2.6.25-rc5         copy_user_generic_string
552309    4.9107  linux-2.6.25-rc5         __slab_free
514792    4.5771  linux-2.6.25-rc5         sock_alloc_send_skb
500879    4.4534  linux-2.6.25-rc5         unix_stream_recvmsg
274798    2.4433  linux-2.6.25-rc5         __kmalloc_track_caller
230283    2.0475  linux-2.6.25-rc5         kfree
222286    1.9764  linux-2.6.25-rc5         unix_stream_sendmsg
217413    1.9331  linux-2.6.25-rc5         memset_c
211589    1.8813  linux-2.6.25-rc5         kmem_cache_alloc
151500    1.3470  linux-2.6.25-rc5         system_call
132262    1.1760  linux-2.6.25-rc5         sock_def_readable
123130    1.0948  linux-2.6.25-rc5         kmem_cache_free
109518    0.9738  linux-2.6.25-rc5         sock_wfree

On tigerton, if I add "slub_max_order=3 slub_min_objects=16" to kernel boot cmdline, the result is improved significantly and it takes just 1/10 time of the original testing.

Below is the new output of slabino -AD.

Name Objects Alloc Free %Fast

:0000192                  3192 80087199 80084141  92   8
kmalloc-512                773 80016203 80015888  97   9
vm_area_struct            2787   223100   220525  94  17
:0004096                    68   118322   118320  99  98
:0000064                 12215   123575   111669  98  42
:0000128                  4616    53826    49422  97  45
dentry                   12373    49568    37286  95  65
:0000080                 12823    33755    21206  99  97

So kmalloc-512 is the key.

Then, I tested it on stoakley with the same kernel commandline. Improvement is about 50%. One important thing is without the boot parameter, hackbench on stoakey takes only 1/4 time of the one on tigerton. With the boot parameter, hackbench on tigerton is faster than the one on stoakely.

Is it possible to initiate slub_min_objects based on possible cpu number? I mean, cpu_possible_map(). We could calculate slub_min_objects by a formular.

-yanmin

#8 lore raw
Quoted 3 lines · Yanmin

On tigerton, if I add "slub_max_order=3 slub_min_objects=16" to kernel boot cmdline, the result is improved significantly and it takes just 1/10 time of the original testing.

Hmmm... That means the updates to SLUB in mm will fix the regression that you are seeing because we there can use large orders of slabs and fallback for all slab caches. But I am still interested to get to the details of slub behavior on the 16p.

Quoted 1 line

So kmalloc-512 is the key.

Yeah in 2.6.26-rc kmalloc-512 has 8 objects per slab. The mm version increases that with a larger allocation size.

Quoted 9 lines

Then, I tested it on stoakley with the same kernel commandline. Improvement is about 50%. One important thing is without the boot parameter, hackbench on stoakey takes only 1/4 time of the one on tigerton. With the boot parameter, hackbench on tigerton is faster than the one on stoakely.

Is it possible to initiate slub_min_objects based on possible cpu number? I mean, cpu_possible_map(). We could calculate slub_min_objects by a formular.

Hmmm... Interesting. Lets first get the details for 2.6.25-rc. Then we can start toying around with the slub version in mm to configure slub in such a way that we get best results on both machines.

#9 lore raw
Quoted 11 lines · Christoph Lameter
Quoted 3 lines · Yanmin

On tigerton, if I add "slub_max_order=3 slub_min_objects=16" to kernel boot cmdline, the result is improved significantly and it takes just 1/10 time of the original testing.

Hmmm... That means the updates to SLUB in mm will fix the regression that you are seeing because we there can use large orders of slabs and fallback for all slab caches. But I am still interested to get to the details of slub behavior on the 16p.

Quoted 4 lines
Quoted 1 line

So kmalloc-512 is the key.

Yeah in 2.6.26-rc kmalloc-512 has 8 objects per slab. The mm version increases that with a larger allocation size.

Would you like to give me a pointer to the patch? Is it one patch, or many patches?

Quoted 7 lines
Quoted 7 lines

Then, I tested it on stoakley with the same kernel commandline. Improvement is about 50%. One important thing is without the boot parameter, hackbench on stoakey takes only 1/4 time of the one on tigerton. With the boot parameter, hackbench on tigerton is faster than the one on stoakely.

Quoted 7 lines
Quoted 3 lines

Is it possible to initiate slub_min_objects based on possible cpu number? I mean, cpu_possible_map(). We could calculate slub_min_objects by a formular.

Hmmm... Interesting. Lets first get the details for 2.6.25-rc. Then we can start toying around with the slub version in mm to configure slub in such a way that we get best results on both machines.

Boot parameter "slub_max_order=3 slub_min_objects=16" could boost perforamnce both on stoakley and on tigerton.

So should we keep slub_min_objects scalable based on possible cpu number? When a machine has more cpu, it means more processes/threads will run on it and it will take more time when they compete for the same resources. SLAB is such a typical resource.

-yanmin

#10 lore raw
Quoted 3 lines · Yanmin
Quoted 2 lines

Yeah in 2.6.26-rc kmalloc-512 has 8 objects per slab. The mm version increases that with a larger allocation size.

Would you like to give me a pointer to the patch? Is it one patch, or many patches?

If you a git pull on the slab-mm branch off my VM tree on kernel.org then you got all you need. There will be an update in the next days though since some of the data you gave me already suggests a couple of ways that things may be made better.

Quoted 5 lines
Quoted 3 lines

Hmmm... Interesting. Lets first get the details for 2.6.25-rc. Then we can start toying around with the slub version in mm to configure slub in such a way that we get best results on both machines.

Boot parameter "slub_max_order=3 slub_min_objects=16" could boost perforamnce both on stoakley and on tigerton.

Well the current slab-mm tree already does order 4 and min_objects=60 which is probably overkill. Next git push on slab-mm will reduce that to the values you found to be sufficient.

Quoted 4 lines

So should we keep slub_min_objects scalable based on possible cpu number? When a machine has more cpu, it means more processes/threads will run on it and it will take more time when they compete for the same resources. SLAB is such a typical resource.

We would have to do some experiments to see how cpu counts affect multiple benchmarks. If we can establish a consistent benefit from varying these parameters based on processor count then we should do so. There is already one example in mm/vmstat.c how this could be done.

#11 lore raw
In reply to Zhang, Yanmin
Quoted 3 lines · Yanmin
Quoted 2 lines

So block 192 and 512's and very active and their fast free percentage is low.

On my 8-core stoakley, there is no such regression. Below data is after testing.

Ok get the detailed statistics for this configuration as well. Then we can see what kind of slub behavior changes between both configurations.

The 16p is really one node? No strange variances in memory latencies?

#12 lore raw
Quoted 8 lines · Christoph Lameter
Quoted 3 lines · Yanmin
Quoted 2 lines

So block 192 and 512's and very active and their fast free percentage is low.

On my 8-core stoakley, there is no such regression. Below data is after testing.

Ok get the detailed statistics for this configuration as well. Then we can see what kind of slub behavior changes between both configurations.

I did paste such data in a prior email. COpy it below.

On my 8-core stoakley, there is no such regression. Below data is after testing. [root@lkp-st02-x8664 ~]# slabinfo -AD
Name Objects Alloc Free %Fast

:0000192                  3170 80055388 80052280  92   1 
:0000512                   316 80012750 80012466  69   1 
vm_area_struct            2642   194700   192193  94  16 
:0000064                  3846    74468    70820  97  53 
:0004096                    15    69014    69012  98  97 
:0000128                  1447    32920    31541  91   8 
dentry                   13485    33060    19652  92  42 
:0000080                 10639    23377    12953  98  98 
:0000096                  1662    16496    15036  99  94 
:0000832                   232    14422    14203  85  10 
:0000016                  2733    15102    13372  99  14 

I ran it for many times and got the similiar output from slabinfo.

Quoted 2 lines

The 16p is really one node?

Yes. It's a SMP machine.

Quoted 1 line

No strange variances in memory latencies?

No.

#13 lore raw
Quoted 2 lines · Yanmin

On my 8-core stoakley, there is no such regression. Below data is after testing.

I was looking for the details on two slab caches. The comparison of the details statistics is likely very interesting because we will be able to see how the doubling of processor counts affects the internal behavior of slub.

#14 lore raw
Quoted 9 lines · Christoph Lameter
Quoted 2 lines · Yanmin

On my 8-core stoakley, there is no such regression. Below data is after testing.

I was looking for the details on two slab caches. The comparison of the details statistics is likely very interesting because we will be able to see how the doubling of processor counts affects the internal behavior of slub.

I collected more data on 16-p tigerton to try to find the possible relationship between slub_min_objects and processor number. Kernel is 2.6.25-rc5.

Command\slub_min_objects | slub_min_objects=8 | 16 | 32 | 64
-------------------------------------------------------------------------------------
./hackbench 100 process 2000 | 250second | 23 | 18.6 | 17.5
./hackbench 200 process 2000 | 532 | 44 | 35.6 | 33.5

The first command line will start 4000 processes and the second will start 8000 processes.

As the problemtic slab is kmalloc-512, slub_min_objects=8 is just the default configuration.

Oprofile data shows the ratio of __slab_alloc+__slab_free+add_partial has no difference between the 2 commandline with the same kernel boot parameters.

slub_min_objects | 8 | 16 | 32 | 64
--------------------------------------------------------------------------------------------
slab(__slab_alloc+__slab_free+add_partial) cpu utilization | 88.00% | 44.00% | 13.00% | 12%

When slub_min_objects=32, we could get a reasonable value. Beyond 32, the improvement is very small. 32 is just possible_cpu_number*2 on my tigerton.

It's hard to say hackbench simulates real applications closely. But it discloses a possible performance bottlebeck. Last year, we once captured the kmalloc-2048 issue by tbench. So the default slub_min_objects need to be revised. In the other hand, slab is allocated by alloc_page when its size is equal to or more than a half page, so enlarging slub_min_objects won't create too many slab page buffers.

As for NUMA, perhaps we could define slub_min_objects to 2*max_cpu_number_per_node.

-yanmin

#15 lore raw
Quoted 7 lines · Yanmin

slub_min_objects | 8 | 16 | 32 | 64
--------------------------------------------------------------------------------------------
slab(__slab_alloc+__slab_free+add_partial) cpu utilization | 88.00% | 44.00% | 13.00% | 12%

When slub_min_objects=32, we could get a reasonable value. Beyond 32, the improvement is very small. 32 is just possible_cpu_number*2 on my tigerton.

Interesting. What is the optimal configuration for your 8p? Could you figure out the optimal configuration for an 4p and a 2p configuration?

Quoted 7 lines

It's hard to say hackbench simulates real applications closely. But it discloses a possible performance bottlebeck. Last year, we once captured the kmalloc-2048 issue by tbench. So the default slub_min_objects need to be revised. In the other hand, slab is allocated by alloc_page when its size is equal to or more than a half page, so enlarging slub_min_objects won't create too many slab page buffers.

As for NUMA, perhaps we could define slub_min_objects to 2*max_cpu_number_per_node.

Well for a 4k cpu configu this would set min_objects to 8192. So I think we could implement a form of logarithmic scaling based on cpu counts comparable to what is done for the statistics update in vmstat.c

fls(num_online_cpus()) = 4

So maybe

slub_min_objects= 8 + (2 + fls(num_online_cpus())) * 4

#16 lore raw
Quoted 7 lines · Christoph Lameter
Quoted 5 lines · Yanmin

slub_min_objects | 8 | 16 | 32 | 64
--------------------------------------------------------------------------------------------
slab(__slab_alloc+__slab_free+add_partial) cpu utilization | 88.00% | 44.00% | 13.00% | 12%

Quoted 5 lines
Quoted 2 lines

When slub_min_objects=32, we could get a reasonable value. Beyond 32, the improvement is very small. 32 is just possible_cpu_number*2 on my tigerton.

Interesting. What is the optimal configuration for your 8p? Could you figure out the optimal configuration for an 4p and a 2p configuration?

I used 8-core stoakley to do testing, and tried boot kernel with maxcpus=4 and 2.

Just ran ./hackbench 100 process 2000.

processor number\slub_min_objects | slub_min_objects=8 | 16 | 32 | 64
--------------------------------------------------------------------------------------------
8p | 60second | 30 | 28.5 | 26.5
--------------------------------------------------------------------------------------------
4p | 50second | 43 | 42 |
--------------------------------------------------------------------------------------------
2p | 92second | 79 | |

As stoakley is just multi-core machine and hasn't hyper-threading, I also tested it on an old harwich machine which has 4 physical processors and 8 logical processors with hyperthreading.
processor number\slub_min_objects | slub_min_objects=8 | 16 | 32 | 64
--------------------------------------------------------------------------------------------
8p | 78.7second | 77.5| |

Quoted 7 lines
Quoted 7 lines

It's hard to say hackbench simulates real applications closely. But it discloses a possible performance bottlebeck. Last year, we once captured the kmalloc-2048 issue by tbench. So the default slub_min_objects need to be revised. In the other hand, slab is allocated by alloc_page when its size is equal to or more than a half page, so enlarging slub_min_objects won't create too many slab page buffers.

Quoted 8 lines
Quoted 1 line

As for NUMA, perhaps we could define slub_min_objects to 2*max_cpu_number_per_node.

Well for a 4k cpu configu this would set min_objects to 8192.
So I think
we could implement a form of logarithmic scaling based on cpu counts comparable to what is done for the statistics update in vmstat.c

fls(num_online_cpus()) = 4

num_online_cpus as the input parameter is ok. A potential issue is how to consider cpu hot-plug.

When num_online_cpus()=16, fls(num_online_cpus())=5.

Quoted 4 lines

So maybe

slub_min_objects= 8 + (2 + fls(num_online_cpus())) * 4

So slub_min_objects= 8 + (1 + fls(num_online_cpus())) * 4.

#17 lore raw
Quoted 1 line · Yanmin

num_online_cpus as the input parameter is ok. A potential issue is how to consider cpu hot-plug.

Yeah I used nr_cpu_ids instead in the patchset that I cced you on. Maybe continue discussion on that thread?

#18 lore raw
In reply to Zhang, Yanmin
Quoted 4 lines · Yanmin

After testing:
Name Objects Alloc Free %Fast

:0000192                  3428 80093958 80090708  92   8
:0000512                   374 80016030 80015715  68   7

Ahhh... Okay those slabs did not change for 2.6.25-rc. Is there really a difference to 2.6.24?

Quoted 1 line

So block 192 and 512's and very active and their fast free percentage is low.

Yes but that is to be expected given that hackbench does allocate objects and then passes them to other processors for freeing.

Could you get me more details on the two critical slabs?

Do slabinfo -a and then pick one alias for each of those sizes.

Then do

slabinfo skbuff_head (whatever alias you want to use to refer to the slab)

for each of them. Should give some more insight as to how slub behaves with these two slab caches.

#19 lore raw
Quoted 9 lines · Christoph Lameter
Quoted 4 lines · Yanmin

After testing:
Name Objects Alloc Free %Fast

:0000192                  3428 80093958 80090708  92   8
:0000512                   374 80016030 80015715  68   7

Ahhh... Okay those slabs did not change for 2.6.25-rc. Is there really a difference to 2.6.24?

As oprofile shows slub functions spend more than 80% cpu time, I would like to focus on optimizing SLUB before going back to 2.6.24.

Quoted 7 lines
Quoted 2 lines

So block 192 and 512's and very active and their fast free percentage is low.

Yes but that is to be expected given that hackbench does allocate objects and then passes them to other processors for freeing.

Could you get me more details on the two critical slabs?

Yes, definitely.

Quoted 2 lines

Do slabinfo -a and then pick one alias for each of those sizes.

They are skbuff_head_cache and kmalloc-512.

Quoted 4 lines

Then do

slabinfo skbuff_head (whatever alias you want to use to refer to the slab)

Slabcache: skbuff_head_cache Aliases: 7 Order : 0 Objects: 2848

Sizes (bytes) Slabs Debug Memory
------------------------------------------------------------------------

Object :     192  Total  :     142   Sanity Checks : Off  Total:  581632
SlabObj:     192  Full   :     126   Redzoning     : Off  Used :  546816
SlabSiz:    4096  Partial:       0   Poisoning     : Off  Loss :   34816
Loss   :       0  CpuSlab:      16   Tracking      : Off  Lalig:       0
Align  :       8  Objects:      21   Tracing       : Off  Lpadd:    9088

skbuff_head_cache has no kmem_cache operations

skbuff_head_cache: Kernel object allocation
-----------------------------------------------------------------------
No Data

skbuff_head_cache: Kernel object freeing
------------------------------------------------------------------------
No Data

skbuff_head_cache: No NUMA information available.

Slab Perf Counter Alloc Free %Al %Fr
--------------------------------------------------
Fastpath 74048234 6259131 92 7

Slowpath              6031994 73818377   7  92
Page Alloc              19746    19603   0   0
Add partial                 0  4658709   0   5
Remove partial        4639106    19603   5   0
RemoteObj/SlabFrozen        0  3887872   0   4

Total 80080228 80077508

Refill 6031979
Deactivate Full=4658836(100%) Empty=0(0%) ToHead=0(0%) ToTail=0(0%)

Slabcache: kmalloc-512 Aliases: 1 Order : 0 Objects: 365

Sizes (bytes) Slabs Debug Memory
------------------------------------------------------------------------

Object :     512  Total  :      61   Sanity Checks : Off  Total:  249856
SlabObj:     512  Full   :      36   Redzoning     : Off  Used :  186880
SlabSiz:    4096  Partial:       9   Poisoning     : Off  Loss :   62976
Loss   :       0  CpuSlab:      16   Tracking      : Off  Lalig:       0
Align  :       8  Objects:       8   Tracing       : Off  Lpadd:       0

kmalloc-512 has no kmem_cache operations

kmalloc-512: Kernel object allocation
-----------------------------------------------------------------------
No Data

kmalloc-512: Kernel object freeing
------------------------------------------------------------------------
No Data

kmalloc-512: No NUMA information available.

Slab Perf Counter Alloc Free %Al %Fr
--------------------------------------------------

Fastpath             55039159  5006829  68   6
Slowpath             24975754 75007769  31  93

Page Alloc 73840 73779 0 0
Add partial 0 24341085 0 30
Remove partial 24267297 73779 30 0
RemoteObj/SlabFrozen 0 953614 0 1
Total 80014913 80014598

Refill 24975738
Deactivate Full=24341121(100%) Empty=0(0%) ToHead=0(0%) ToTail=0(0%)

Quoted 4 lines

for each of them. Should give some more insight as to how slub behaves with these two slab caches.

#20 lore raw
Quoted 4 lines · Yanmin
Quoted 2 lines

Ahhh... Okay those slabs did not change for 2.6.25-rc. Is there really a difference to 2.6.24?

As oprofile shows slub functions spend more than 80% cpu time, I would like to focus on optimizing SLUB before going back to 2.6.24.

I thought you wanted to address a regression vs 2.6.24?

Quoted 9 lines

kmalloc-512: No NUMA information available.

Slab Perf Counter Alloc Free %Al %Fr
--------------------------------------------------

Fastpath             55039159  5006829  68   6
Slowpath             24975754 75007769  31  93

Page Alloc 73840 73779 0 0
Add partial 0 24341085 0 30
Remove partial 24267297 73779 30 0

^^^ add partial/remove partial is likely the cause for trouble here. 30% is unacceptably high. The larger allocs will reduce the partial handling overhead. That is likely the effect that we see here.

Quoted 1 line

Refill 24975738

Duh refills at 50%? We could try to just switch to another slab instead of reusing the existing one. May also affect the add/remove partial situation.

#21 lore raw

Here is a patch to just not perform refills but switch slabs instead. Could check what effect doing so has on the statistics you see on the 16p?

---
 mm/slub.c |    5 +----
 1 file changed, 1 insertion(+), 4 deletions(-)
mm/slub.c +1 -4
Index: linux-2.6/mm/slub.c
===================================================================
@@ -1474,10 +1474,7 @@ static void *__slab_alloc(struct kmem_ca
 		goto new_slab;
 
 	slab_lock(c->page);
-	if (unlikely(!node_match(c, node)))
-		goto another_slab;
-
-	stat(c, ALLOC_REFILL);
+	goto another_slab;
 
 load_freelist:
 	object = c->page->freelist;
#22 lore raw
Quoted 13 lines · Christoph Lameter

Here is a patch to just not perform refills but switch slabs instead. Could check what effect doing so has on the statistics you see on the 16p?

---
 mm/slub.c |    5 +----
 1 file changed, 1 insertion(+), 4 deletions(-)

Index: linux-2.6/mm/slub.c
===================================================================

mm/slub.c +0 -0
@@ -1474,10 +1474,7 @@ static void *__slab_alloc(struct kmem_ca
 		goto new_slab;

mm/slub.c static void *__slab_alloc(struct kmem_ca · line 1480

Quoted 10 lines
mm/slub.c +1 -4
 
 	slab_lock(c->page);
-	if (unlikely(!node_match(c, node)))
-		goto another_slab;
-
-	stat(c, ALLOC_REFILL);
+	goto another_slab;
 
 load_freelist:
 	object = c->page->freelist;

There is no much help. In 2.6.25-rc5, REFILL means refill from c->page->freelist and another_slab. It's looks like its definition is confusing. In the case of hackbench, mostly, c->page->freelist is NULL.

With #hackbench 100 process 2000, 100*20*2 (totoally 4000) processes are started. vmstat shows about 300~500 processes are at RUNNING state, so every processor runqueue has more than 20 processes running on 16p tigerton.

Below is the data with kernel 2.6.25-rc5+your_patch.

[ymzhang@lkp-tt01-x8664 ~]$ slabinfo kmalloc-512

Slabcache: kmalloc-512 Aliases: 1 Order : 0 Objects: 352

Sizes (bytes) Slabs Debug Memory
------------------------------------------------------------------------

Object :     512  Total  :      56   Sanity Checks : Off  Total:  229376
SlabObj:     512  Full   :      36   Redzoning     : Off  Used :  180224
SlabSiz:    4096  Partial:       4   Poisoning     : Off  Loss :   49152
Loss   :       0  CpuSlab:      16   Tracking      : Off  Lalig:       0
Align  :       8  Objects:       8   Tracing       : Off  Lpadd:       0

kmalloc-512 has no kmem_cache operations

kmalloc-512: Kernel object allocation
-----------------------------------------------------------------------
No Data

kmalloc-512: Kernel object freeing
------------------------------------------------------------------------
No Data

kmalloc-512: No NUMA information available.

Slab Perf Counter Alloc Free %Al %Fr
--------------------------------------------------

Fastpath             55883575  6130576  69   7
Slowpath             24131134 73883818  30  92

Page Alloc 84844 84788 0 0
Add partial 270625 23860257 0 29
Remove partial 24046290 84752 30 0
RemoteObj/SlabFrozen 270825 439015 0 0
Total 80014709 80014394

Deactivate Full=23860293(98%) Empty=200(0%) ToHead=0(0%) ToTail=270625(1%)

#23 lore raw
Quoted 3 lines · Yanmin

There is no much help. In 2.6.25-rc5, REFILL means refill from c->page->freelist and another_slab. It's looks like its definition is confusing. In the case of hackbench, mostly, c->page->freelist is NULL.

REFILL means refilling the per cpu objects from the freelist of the per cpu slab page. That could be bad because it requires taking the slab lock on the slab page.

Quoted 7 lines

Slab Perf Counter Alloc Free %Al %Fr
--------------------------------------------------

Fastpath             55883575  6130576  69   7
Slowpath             24131134 73883818  30  92

Page Alloc 84844 84788 0 0
Add partial 270625 23860257 0 29
Remove partial 24046290 84752 30 0

Hmmm... I was hoping that add/remove partial numbers would come down. Ok lets forget about the patch. Increasing min_objects does the trick.

#24 lore raw
In reply to Christoph Lameter
Quoted 8 lines · Christoph Lameter
Quoted 4 lines · Yanmin
Quoted 2 lines

Ahhh... Okay those slabs did not change for 2.6.25-rc. Is there really a difference to 2.6.24?

As oprofile shows slub functions spend more than 80% cpu time, I would like to focus on optimizing SLUB before going back to 2.6.24.

I thought you wanted to address a regression vs 2.6.24?

Initially I wanted to do so, but oprofile data showed both 2.6.24 and 2.6.25-rc aren't good with hachbench on tigerton.

The slub_min_objects boot parameter could boost performance largely. So I think we need optimize it before addressing the regression.

Quoted 23 lines
Quoted 10 lines

kmalloc-512: No NUMA information available.

Slab Perf Counter Alloc Free %Al %Fr
--------------------------------------------------

Fastpath             55039159  5006829  68   6
Slowpath             24975754 75007769  31  93

Page Alloc 73840 73779 0 0
Add partial 0 24341085 0 30
Remove partial 24267297 73779 30 0

^^^ add partial/remove partial is likely the cause for trouble here. 30% is unacceptably high. The larger allocs will reduce the partial handling overhead. That is likely the effect that we see here.

Quoted 1 line

Refill 24975738

Duh refills at 50%? We could try to just switch to another slab instead of reusing the existing one. May also affect the add/remove partial situation.

#25 lore raw
In reply to Zhang, Yanmin
Quoted 8 lines · Yanmin

Comparing with 2.6.24, on my 16-core tigerton, hackbench process mode has about 40% regression with 2.6.25-rc1, and more than 20% regression with kernel 2.6.25-rc4, because rc4 includes the reverting patch of scheduler load balance.

Command to start it.
#hackbench 100 process 2000
I ran it for 3 times and sum the values.

Quoted 6 lines

I tried to investiagte it by bisect.
Kernel up to tag 0f4dafc0563c6c49e17fe14b3f5f356e4c4b8806 has the 20% regression. Kernel up to tag 6e90aa972dda8ef86155eefcdbdc8d34165b9f39 hasn't regression.

Any bisect between above 2 tags cause kernel hang. I tried to checkout to a point between these 2 tags for many times manually and kernel always paniced.

Where is the kernel panicing? The changeset right after the last one
above: bc87d2fe7a1190f1c257af8a91fc490b1ee35954, is a change to efivars, are you using that in your .config?

Quoted 2 lines

All patches between the 2 tags are on kobject restructure. I guess such restructure creates more cache miss on the 16-core tigerton.

Nothing should be creating kobjects on a normal load like this, so a regression seems very odd. Unless the /sys/kernel/uids/ stuff is triggering this?

Do you have a link to where I can get hackbench (google seems to find lots of reports with it, but not the source itself), so I can test to see if we are accidentally creating kobjects with this load?

thanks,

greg k-h

#26 lore raw
Quoted 23 lines · On Thu, 13 Mar 2008 08:14:13 -0700 Greg KH wrote:
Quoted 8 lines · Yanmin

Comparing with 2.6.24, on my 16-core tigerton, hackbench process mode has about 40% regression with 2.6.25-rc1, and more than 20% regression with kernel 2.6.25-rc4, because rc4 includes the reverting patch of scheduler load balance.

Command to start it.
#hackbench 100 process 2000
I ran it for 3 times and sum the values.

Quoted 6 lines

I tried to investiagte it by bisect.
Kernel up to tag 0f4dafc0563c6c49e17fe14b3f5f356e4c4b8806 has the 20% regression. Kernel up to tag 6e90aa972dda8ef86155eefcdbdc8d34165b9f39 hasn't regression.

Any bisect between above 2 tags cause kernel hang. I tried to checkout to a point between these 2 tags for many times manually and kernel always paniced.

Where is the kernel panicing? The changeset right after the last one
above: bc87d2fe7a1190f1c257af8a91fc490b1ee35954, is a change to efivars, are you using that in your .config?

Quoted 2 lines

All patches between the 2 tags are on kobject restructure. I guess such restructure creates more cache miss on the 16-core tigerton.

Quoted 7 lines

Nothing should be creating kobjects on a normal load like this, so a regression seems very odd. Unless the /sys/kernel/uids/ stuff is triggering this?

Do you have a link to where I can get hackbench (google seems to find lots of reports with it, but not the source itself), so I can test to see if we are accidentally creating kobjects with this load?

The version that I see referenced most often (unscientifically :) is somewhere under people.redhat.com/mingo/, like so:
http://people.redhat.com/mingo/cfs-scheduler/tools/hackbench.c

---
~Randy

#27 lore raw
Quoted 29 lines · Randy Dunlap
Quoted 27 lines · On Thu, 13 Mar 2008 08:14:13 -0700 Greg KH wrote:
Quoted 8 lines · Yanmin

Comparing with 2.6.24, on my 16-core tigerton, hackbench process mode has about 40% regression with 2.6.25-rc1, and more than 20% regression with kernel 2.6.25-rc4, because rc4 includes the reverting patch of scheduler load balance.

Command to start it.
#hackbench 100 process 2000
I ran it for 3 times and sum the values.

Quoted 6 lines

I tried to investiagte it by bisect.
Kernel up to tag 0f4dafc0563c6c49e17fe14b3f5f356e4c4b8806 has the 20% regression. Kernel up to tag 6e90aa972dda8ef86155eefcdbdc8d34165b9f39 hasn't regression.

Any bisect between above 2 tags cause kernel hang. I tried to checkout to a point between these 2 tags for many times manually and kernel always paniced.

Where is the kernel panicing? The changeset right after the last one
above: bc87d2fe7a1190f1c257af8a91fc490b1ee35954, is a change to efivars, are you using that in your .config?

Quoted 2 lines

All patches between the 2 tags are on kobject restructure. I guess such restructure creates more cache miss on the 16-core tigerton.

Nothing should be creating kobjects on a normal load like this, so a regression seems very odd. Unless the /sys/kernel/uids/ stuff is triggering this?

Quoted 7 lines
Quoted 3 lines

Do you have a link to where I can get hackbench (google seems to find lots of reports with it, but not the source itself), so I can test to see if we are accidentally creating kobjects with this load?

The version that I see referenced most often (unscientifically :) is somewhere under people.redhat.com/mingo/, like so:
http://people.redhat.com/mingo/cfs-scheduler/tools/hackbench.c

Great, thanks for the link.

In using that version, I do not see any kobjects being created at all when running the program. So I don't see how a kobject change could have caused any slowdown.

Yanmin, is the above link the version you are using?

Hm, running with "hackbench 100 process 2000" seems to lock up my laptop, maybe I shouldn't run 4000 tasks at once on such a memory starved machine...

thanks,

greg k-h

#28 lore raw
Quoted 38 lines · Greg KH
Quoted 29 lines · Randy Dunlap
Quoted 27 lines · On Thu, 13 Mar 2008 08:14:13 -0700 Greg KH wrote:
Quoted 8 lines · Yanmin

Comparing with 2.6.24, on my 16-core tigerton, hackbench process mode has about 40% regression with 2.6.25-rc1, and more than 20% regression with kernel 2.6.25-rc4, because rc4 includes the reverting patch of scheduler load balance.

Command to start it.
#hackbench 100 process 2000
I ran it for 3 times and sum the values.

Quoted 6 lines

I tried to investiagte it by bisect.
Kernel up to tag 0f4dafc0563c6c49e17fe14b3f5f356e4c4b8806 has the 20% regression. Kernel up to tag 6e90aa972dda8ef86155eefcdbdc8d34165b9f39 hasn't regression.

Any bisect between above 2 tags cause kernel hang. I tried to checkout to a point between these 2 tags for many times manually and kernel always paniced.

Where is the kernel panicing? The changeset right after the last one
above: bc87d2fe7a1190f1c257af8a91fc490b1ee35954, is a change to efivars, are you using that in your .config?

Quoted 2 lines

All patches between the 2 tags are on kobject restructure. I guess such restructure creates more cache miss on the 16-core tigerton.

Nothing should be creating kobjects on a normal load like this, so a regression seems very odd. Unless the /sys/kernel/uids/ stuff is triggering this?

Quoted 7 lines
Quoted 3 lines

Do you have a link to where I can get hackbench (google seems to find lots of reports with it, but not the source itself), so I can test to see if we are accidentally creating kobjects with this load?

The version that I see referenced most often (unscientifically :) is somewhere under people.redhat.com/mingo/, like so:
http://people.redhat.com/mingo/cfs-scheduler/tools/hackbench.c

Quoted 7 lines

Great, thanks for the link.

In using that version, I do not see any kobjects being created at all when running the program. So I don't see how a kobject change could have caused any slowdown.

Yanmin, is the above link the version you are using?

Yes.

Quoted 4 lines

Hm, running with "hackbench 100 process 2000" seems to lock up my laptop, maybe I shouldn't run 4000 tasks at once on such a memory starved machine...

The issue doesn't exist on my 8-core stoakley and on tulsa. So I don't think you could reproduce it on laptop.

Quoted 1 line

From the oprofile data, perhaps we need dig into SLUB firstly.

-yanmin

#29 lore raw
Quoted 46 lines · Yanmin
Quoted 38 lines · Greg KH
Quoted 29 lines · Randy Dunlap
Quoted 27 lines · On Thu, 13 Mar 2008 08:14:13 -0700 Greg KH wrote:
Quoted 8 lines · Yanmin

Comparing with 2.6.24, on my 16-core tigerton, hackbench process mode has about 40% regression with 2.6.25-rc1, and more than 20% regression with kernel 2.6.25-rc4, because rc4 includes the reverting patch of scheduler load balance.

Command to start it.
#hackbench 100 process 2000
I ran it for 3 times and sum the values.

Quoted 6 lines

I tried to investiagte it by bisect.
Kernel up to tag 0f4dafc0563c6c49e17fe14b3f5f356e4c4b8806 has the 20% regression. Kernel up to tag 6e90aa972dda8ef86155eefcdbdc8d34165b9f39 hasn't regression.

Any bisect between above 2 tags cause kernel hang. I tried to checkout to a point between these 2 tags for many times manually and kernel always paniced.

Where is the kernel panicing? The changeset right after the last one
above: bc87d2fe7a1190f1c257af8a91fc490b1ee35954, is a change to efivars, are you using that in your .config?

Quoted 2 lines

All patches between the 2 tags are on kobject restructure. I guess such restructure creates more cache miss on the 16-core tigerton.

Nothing should be creating kobjects on a normal load like this, so a regression seems very odd. Unless the /sys/kernel/uids/ stuff is triggering this?

Quoted 7 lines
Quoted 3 lines

Do you have a link to where I can get hackbench (google seems to find lots of reports with it, but not the source itself), so I can test to see if we are accidentally creating kobjects with this load?

The version that I see referenced most often (unscientifically :) is somewhere under people.redhat.com/mingo/, like so:
http://people.redhat.com/mingo/cfs-scheduler/tools/hackbench.c

Quoted 7 lines

Great, thanks for the link.

In using that version, I do not see any kobjects being created at all when running the program. So I don't see how a kobject change could have caused any slowdown.

Yanmin, is the above link the version you are using?

Quoted 8 lines

Yes.

Quoted 4 lines

Hm, running with "hackbench 100 process 2000" seems to lock up my laptop, maybe I shouldn't run 4000 tasks at once on such a memory starved machine...

The issue doesn't exist on my 8-core stoakley and on tulsa. So I don't think you could reproduce it on laptop.

But I should see any kobjects being created and destroyed as you are thinking that is the problem here, right?

And I don't see any, so I'm thinking that this is probably something else.

I'm still interested in why your machine was oopsing when bisecting through the kobject commits. I thought it all should have worked without problems, as I spend enough time trying to ensure it was so...

thanks,

greg k-h

#30 lore raw
Quoted 50 lines · Greg KH
Quoted 49 lines · Yanmin
Quoted 38 lines · Greg KH
Quoted 29 lines · Randy Dunlap
Quoted 27 lines · On Thu, 13 Mar 2008 08:14:13 -0700 Greg KH wrote:
Quoted 8 lines · Yanmin

Comparing with 2.6.24, on my 16-core tigerton, hackbench process mode has about 40% regression with 2.6.25-rc1, and more than 20% regression with kernel 2.6.25-rc4, because rc4 includes the reverting patch of scheduler load balance.

Command to start it.
#hackbench 100 process 2000
I ran it for 3 times and sum the values.

Quoted 6 lines

I tried to investiagte it by bisect.
Kernel up to tag 0f4dafc0563c6c49e17fe14b3f5f356e4c4b8806 has the 20% regression. Kernel up to tag 6e90aa972dda8ef86155eefcdbdc8d34165b9f39 hasn't regression.

Any bisect between above 2 tags cause kernel hang. I tried to checkout to a point between these 2 tags for many times manually and kernel always paniced.

Where is the kernel panicing? The changeset right after the last one
above: bc87d2fe7a1190f1c257af8a91fc490b1ee35954, is a change to efivars, are you using that in your .config?

Quoted 2 lines

All patches between the 2 tags are on kobject restructure. I guess such restructure creates more cache miss on the 16-core tigerton.

Nothing should be creating kobjects on a normal load like this, so a regression seems very odd. Unless the /sys/kernel/uids/ stuff is triggering this?

Quoted 7 lines
Quoted 3 lines

Do you have a link to where I can get hackbench (google seems to find lots of reports with it, but not the source itself), so I can test to see if we are accidentally creating kobjects with this load?

The version that I see referenced most often (unscientifically :) is somewhere under people.redhat.com/mingo/, like so:
http://people.redhat.com/mingo/cfs-scheduler/tools/hackbench.c

Quoted 7 lines

Great, thanks for the link.

In using that version, I do not see any kobjects being created at all when running the program. So I don't see how a kobject change could have caused any slowdown.

Yanmin, is the above link the version you are using?

Yes.

Quoted 1 line
Quoted 8 lines
Quoted 5 lines
Quoted 3 lines

Hm, running with "hackbench 100 process 2000" seems to lock up my laptop, maybe I shouldn't run 4000 tasks at once on such a memory starved machine...

The issue doesn't exist on my 8-core stoakley and on tulsa. So I don't think you could reproduce it on laptop.

But I should see any kobjects being created and destroyed as you are thinking that is the problem here, right?

Not just thinking. That's based on lots of testing. But as you know, performance work is complicated often. Now, I think maybe kernel image changes cache line alignment.

Quoted 3 lines

And I don't see any, so I'm thinking that this is probably something else.

Yes.

Quoted 4 lines

I'm still interested in why your machine was oopsing when bisecting through the kobject commits. I thought it all should have worked without problems, as I spend enough time trying to ensure it was so...

Kernel panic after printing warning in kref_get when executing add_disk

in rd_init.

Thanks,
Yanmin