slub: Make the order configurable for each slab cache

Makes /sys/kernel/slab/<slabname>/order writable. The allocation order of a slab cache can then be changed dynamically during runtime.

Signed-off-by Christoph Lameter <[email protected]>

(quoted diff) +1 -0

@@ -3715,11 +3720,23 @@ static ssize_t objs_per_slab_show(struct
 }
 SLAB_ATTR_RO(objs_per_slab);
 
+static ssize_t order_store(struct kmem_cache *s,

I think we either want to check that the order is big enough to hold one object for the given cache or add a comment explaining why it can never happen (page allocator pass-through).

Quoted 3 lines

I think we either want to check that the order is big enough to hold one object for the given cache or add a comment explaining why it can never happen (page allocator pass-through).

Calculate_sizes() will violate max_order if the object does not fit.

I am not sure I understand what you mean here. For example, for a cache that requires minimum order of 1 to fit any objects (which doesn't happen now because of page allocator pass-through), the order_store() function can call calculate_sizes() with forced_order

set to zero after which the cache becomes useless. That deserves a code comment, I think.

Quoted 4 lines · Pekka Enberg

I am not sure I understand what you mean here. For example, for a cache that requires minimum order of 1 to fit any objects (which doesn't happen now because of page allocator pass-through), the order_store() function can call calculate_sizes() with forced_order

It does happen because the page allocator pass through is only possible for kmalloc allocations.

Quoted 2 lines

set to zero after which the cache becomes useless. That deserves a code comment, I think.

If the object does not fit into a page then calculate_sizes will violate max_order (if necessary) in order to make sure that an allocation is possible.

Hmm, I seem to be missing something here. For page size of 4KB, object size of 8KB, and min_order of zero, when I write zero order to /sys/kernel/slab/<cache>/order the kernel won't crash because...?

Since we now have total_objects counter per node use that to check for the presence of any objects.

Signed-off-by Christoph Lameter <[email protected]>

Since there is now no worry anymore about higher order allocs (hopefully) increase the minimum of objects per slab to 60 so that slub can reach a similar fastpath/slowpath ratio as slab. Set the max order to default to 4 (64k) and require slub to use a higher order if a certain object density cannot be reached.

Signed-off-by Christoph Lameter <[email protected]>

I can see why you want to change the defaults for big iron but why not keep the existing PAGE_SHIFT check which leaves embedded and regular desktop unchanged?

Quoted 3 lines

I can see why you want to change the defaults for big iron but why not keep the existing PAGE_SHIFT check which leaves embedded and regular desktop unchanged?

The defaults for slab are also 60 objects per slab. The PAGE_SHIFT says nothing about the big iron. Our new big irons have a page shift of 12 and are x86_64.

We could drop the limit if CONFIG_EMBEDDED is set but then this may waste space. A higher order allows slub to reach a higher object density (in particular for objects 500-2000 bytes size).

Quoted 3 lines · Christoph Lameter

The defaults for slab are also 60 objects per slab. The PAGE_SHIFT says nothing about the big iron. Our new big irons have a page shift of 12 and are x86_64.

Where is that objects per slab limit? I only see calculate_slab_order() trying out bunch of page orders until we hit "acceptable" internal fragmentation. Also keep in mind how badly SLAB compares to SLUB and SLOB in terms of memory efficiency.

Quoted 3 lines

We could drop the limit if CONFIG_EMBEDDED is set but then this may waste space. A higher order allows slub to reach a higher object density (in particular for objects 500-2000 bytes size).

I am more worried about memory allocated for objects that are not used rather than memory wasted due to bad fitting.

Quoted 9 lines · Pekka J Enberg

Quoted 3 lines · Christoph Lameter

The defaults for slab are also 60 objects per slab. The PAGE_SHIFT says nothing about the big iron. Our new big irons have a page shift of 12 and are x86_64.

Where is that objects per slab limit? I only see calculate_slab_order() trying out bunch of page orders until we hit "acceptable" internal fragmentation. Also keep in mind how badly SLAB compares to SLUB and SLOB in terms of memory efficiency.

slub_min_objects sets that limit.

Quoted 5 lines · Christoph Lameter

Quoted 3 lines

We could drop the limit if CONFIG_EMBEDDED is set but then this may waste space. A higher order allows slub to reach a higher object density (in particular for objects 500-2000 bytes size).

Quoted 2 lines

I am more worried about memory allocated for objects that are not used rather than memory wasted due to bad fitting.

Is there any way to quantify this? This is likely only an effect that mostly matters for rarely used slabs (the merging reduces that effect). F.e. fitting more inodes or dentries into a single slab increases object density.

On the other hand, a single object can now pin 64k in memory rather than 4k. So when we collapse some cache under memory pressure, we're not likely to free as much.

I know you've put a lot of effort into dealing with the dcache and icache instances of this, but this could very well offset most of that.

Also, we might consider only allocating an order-1 slab if we've filled an order-0, and so on. When we hit pressure, we kick our order counter back to 0.

Change the statistics to consider that slabs of the same slabcache can have different number of objects in them since they may be of different order.

Provide a new sysfs field

total_objects

which shows the total objects that the allocated slabs of a slabcache could hold.

Update the description of the objects field in the kmem_cache structure. Its role is now to be the limit of the maximum number of objects per slab if a slab is allocated with the largest possible order.

Signed-off-by Christoph Lameter <[email protected]>

Reviewed-by Pekka Enberg <[email protected]>

Hmmm... I get some weird numbers when I use slabinfo but cannot spot the issue. Could you look a bit closer at this? In particular at the slabinfo emulation?

@@ -4331,7 +4367,9 @@ static int s_show(struct seq_file *m, vo

        unsigned long nr_partials = 0;

        unsigned long nr_slabs = 0;
        unsigned long nr_inuse = 0;

 -       unsigned long nr_objs;
 +       unsigned long nr_objs = 0;
 +       unsigned long nr_partial_inuse = 0;
 +       unsigned long nr_partial_total = 0;
        struct kmem_cache *s;
        int node;

@@ -4345,14 +4383,15 @@ static int s_show(struct seq_file *m, vo

                nr_partials += n->nr_partial;
                nr_slabs += atomic_long_read(&n->nr_slabs);

- nr_inuse += count_partial(n);
+ nr_objs += atomic_long_read(&n->total_objects);

+ nr_partial_inuse += count_partial_inuse(n);

 +               nr_partial_total += count_partial_total(s, n);
        }

 -       nr_objs = nr_slabs * s->objects;
 -       nr_inuse += (nr_slabs - nr_partials) * s->objects;
 +       nr_inuse = nr_objs - (nr_partial_total - nr_partial_inuse);

This is the patchset that was posted two weeks ago modified according to the feedback that Pekka gave. I would like to put these patches into mm.

This patchset makes slub capable of handling arbitrary sizes of pages. This means that a slab cache that currently uses order 1 because of packing density issues can fallback to order 0 allocations if memory becomes fragmented. All allocations for objects <= PAGE_SIZE can fall back like that. So a single slab may contain various sizes of pages that may contain more or less objects.

On the other hand it also enables slub to use larger page orders by default since it is now no problem to fall back to an order 0 alloc. The default max order is set to 4 which means that 64K compound pages can beused in some situations for large objects that do not fit into smaller pages. This in turn increases the number of times slub can use its fastpath before a fallback to the page allocator has to occur.

The patchset realizes the initial intend of providing a feature comparable with the per cpu queue size in slab. The order for each slab cache can be configured from user space while the system is running. Increasing the default allocation order can be used to tune slub like slab.

The allocated sizes can then also be effectively controlled via boot parameters (slub_min_order and slub_max_order).

The patchset is also available via git

git://git.kernel.org/pub/scm/linux/kernel/git/christoph/vm.git slab-mm

Loss to Gain
Kernbench Elapsed time -0.64% 0.32%
Kernbench Total time -0.61% 0.48%
Hackbench sockets-12 clients -2.95% 5.13%
Hackbench pipes-12 clients -16.95% 9.27%
TBench 4 clients -1.98% 8.2%
DBench 4 clients (ext2) -5.9% 7.99%

So, running with the high orders is not a clear-cut win to my eyes. What did you test to show that it was a general win justifying a high-order by default? From looking through, tbench seems to be the only obvious one to gain but the rest, it is not clear at all. I'll try give sysbench a spin later to see if it is clear-cut.

However, in *all* cases, superpage allocations were less successful and in some cases it was severely regressed (one machine went from 81% success rate to 36%). Sufficient statistics are not gathered to see why this happened in retrospect but my suspicion would be that high-order RECLAIMABLE and UNMOVABLE slub allocations routinely fall back to the less fragmented MOVABLE pageblocks with these patches - something that is normally a very rare event. This change in assumption hurts fragmentation avoidance and chances are the long-term behaviour of these patches is not great.

If this guess is correct, using a high-order size by default is a bad plan and it should only be set when it is known that the target workload benefits and superpage allocations are not a concern. Alternative, set high-order by default only for a limited number of caches that are RECLAIMABLE (or better yet ones we know can be directly reclaimed with the slub-defrag patches).

As it is, this is painful from a fragmentation perspective and the performance win is not clear-cut.

Quoted 9 lines

Loss to Gain
Kernbench Elapsed time -0.64% 0.32%
Kernbench Total time -0.61% 0.48%
Hackbench sockets-12 clients -2.95% 5.13%
Hackbench pipes-12 clients -16.95% 9.27%
TBench 4 clients -1.98% 8.2%
DBench 4 clients (ext2) -5.9% 7.99%

Quoted 5 lines

So, running with the high orders is not a clear-cut win to my eyes. What did you test to show that it was a general win justifying a high-order by default? From looking through, tbench seems to be the only obvious one to gain but the rest, it is not clear at all. I'll try give sysbench a spin later to see if it is clear-cut.

Hmmm... Interesting. The tests that I did awhile ago were with max order
3. The patch as is now has max order 4. Maybe we need to reduce the order?

Looks like this was mostly a gain except for hackbench. Which is to be expected since the benchmark shelves out objects from the same slab round robin to different cpus. The higher the number of objects in the slab the higher the chance of contention on the slab lock.

Quoted 8 lines

However, in *all* cases, superpage allocations were less successful and in some cases it was severely regressed (one machine went from 81% success rate to 36%). Sufficient statistics are not gathered to see why this happened in retrospect but my suspicion would be that high-order RECLAIMABLE and UNMOVABLE slub allocations routinely fall back to the less fragmented MOVABLE pageblocks with these patches - something that is normally a very rare event. This change in assumption hurts fragmentation avoidance and chances are the long-term behaviour of these patches is not great.

Superpage allocations means huge page allocations?

Enable slub statistics
and you will be able to see the number of fallbacks in /sys/kernel/slab/xx/order_fallback to confirm your suspicions.

How would the allocator be able to get MOVABLE allocations? Is fallback permitted for order 0 allocs to MOVABLE?

Quoted 8 lines

If this guess is correct, using a high-order size by default is a bad plan and it should only be set when it is known that the target workload benefits and superpage allocations are not a concern. Alternative, set high-order by default only for a limited number of caches that are RECLAIMABLE (or better yet ones we know can be directly reclaimed with the slub-defrag patches).

As it is, this is painful from a fragmentation perspective and the performance win is not clear-cut.

Could we reduce the max order to 3 and see what happens then?

Ok, I'm offically a tool. I had named patchsets wrong and tested slub-defrag instead of slub-highorder. I didn't notice until I opened the diff file to set the max_order. slub-highorder is being tested at the moment but it'll be hours before it completes.

Quoted 5 lines

Ok, I'm offically a tool. I had named patchsets wrong and tested slub-defrag instead of slub-highorder. I didn't notice until I opened the diff file to set the max_order. slub-highorder is being tested at the moment but it'll be hours before it completes.

Tool? Never heard it before. Is that an Irish term?

Do not worry. That
happens all the time in the computer industry. These days, I get suspicious when people claim something is perfect (100% yes!).

For huge page allocation success rates, the high order never helper the situation but it was nowhere near as severe as it was for the slub-defrag patches (ironically enough). Only one machine showed significantly worse

results. The rest were comparable for this set of tests at least but I would still be wary of the long-lived behaviour of high-order slab allocations slowly fragmenting memory due to pageblock fallbacks. Will think of how to prove that in some way but just re-running the tests multiple times without reboot may be enough.

Setting the order to 3 had vaguely similar results. The two outlier machines had even worse negatives than order-4. With those machines omitted the results were

Same story, hackbench-pipes and dbench suffer badly on some machines. It's a similar story for order-1. With machine omitted it's

Kernbench Elapsed    time      -0.14      to 0.24
Kernbench Total  CPU           -0.13      to 0.11
Hackbench pipes-1              -11.90     to 5.39
Hackbench pipes-4              -7.01      to 2.06
Hackbench pipes-8              -5.49      to 1.66
Hackbench pipes-16             -6.08      to 2.72
Hackbench sockets-1            0.28       to 6.99
Hackbench sockets-4            0.63       to 5.50
Hackbench sockets-8            -10.95     to 7.70
Hackbench sockets-16           0.64       to 12.16

TBench    clients-1            -3.94      to 1.05
TBench    clients-2            -11.96     to 3.25
TBench    clients-4            -12.48     to -1.12
TBench    clients-8            -11.82     to -8.56
DBench    clients-1-ext2       -12.20     to 2.27
DBench    clients-2-ext2       -4.23      to 0.57
DBench    clients-4-ext2       -2.31      to 3.96
DBench    clients-8-ext2       -3.65      to 6.09

Based on this set of tests, it's clear that raising the order can be a big win but setting it as default is less clear-cut.

Quoted 4 lines

For huge page allocation success rates, the high order never helper the situation but it was nowhere near as severe as it was for the slub-defrag patches (ironically enough). Only one machine showed significantly worse

Well the slub-defrag tree is not really in shape for testing at this point and I was working on it the last week. So not sure what tree was picked up and thus not sure what to deduce from it. It may be too aggressive in defragmentation attempts.

Quoted 5 lines

results. The rest were comparable for this set of tests at least but I would still be wary of the long-lived behaviour of high-order slab allocations slowly fragmenting memory due to pageblock fallbacks. Will think of how to prove that in some way but just re-running the tests multiple times without reboot may be enough.

Well maybe we could tune the page allocator a bit? There is the order 0 issue. We could also make all slab allocations use the same slab order in order to reduce fragmentation problems.

Quoted 3 lines

Setting the order to 3 had vaguely similar results. The two outlier machines had even worse negatives than order-4. With those machines omitted the results were

Wonder what made them go worse.

Quoted 13 lines

Same story, hackbench-pipes and dbench suffer badly on some machines. It's a similar story for order-1. With machine omitted it's

Kernbench Elapsed    time      -0.14      to 0.24
Kernbench Total  CPU           -0.13      to 0.11
Hackbench pipes-1              -11.90     to 5.39
Hackbench pipes-4              -7.01      to 2.06
Hackbench pipes-8              -5.49      to 1.66
Hackbench pipes-16             -6.08      to 2.72
Hackbench sockets-1            0.28       to 6.99
Hackbench sockets-4            0.63       to 5.50
Hackbench sockets-8            -10.95     to 7.70
Hackbench sockets-16           0.64       to 12.16

Quoted 8 lines

TBench    clients-1            -3.94      to 1.05
TBench    clients-2            -11.96     to 3.25
TBench    clients-4            -12.48     to -1.12
TBench    clients-8            -11.82     to -8.56
DBench    clients-1-ext2       -12.20     to 2.27
DBench    clients-2-ext2       -4.23      to 0.57
DBench    clients-4-ext2       -2.31      to 3.96
DBench    clients-8-ext2       -3.65      to 6.09

Well in that case there is something going on very strange performance wise. The results should be equal to upstream since the same orders are used.

The only change in the hotpaths is another lookup which cannot really account for the variances we see here. An 12% improvement because logic was added to the hotpath?

There should be a significant regression
tbench (2%-4%) because the 4k slab cache must cause trouble.

Quoted 2 lines

Based on this set of tests, it's clear that raising the order can be a big win but setting it as default is less clear-cut.

There is something wrong here and we need to figure out what it is. The order-1 test should fairly accurately reproduce upstream performance characteristics.

I don't think it would reduce them unless everyone was always using the same order. Once slub is using a higher order than everywhere else, it is possible it will use an alternative pageblock type just for the high order.

The only tuning of the page allocator I can think of is to teach rmqueue_bulk() to use the fewer high-order allocations to batch refill the pcp queues. It's not very straight-forward though as when I tried this a bit over a year ago, it cause fragmentation problems of its own. I'll see about trying again.

Quoted 3 lines

Well in that case there is something going on very strange performance wise. The results should be equal to upstream since the same orders are used.

Really, order-1 is used by default by SLUB upstream? I missed that and it doesn't appear to be the case on 2.6.25-rc2-mm1 at least according to slabinfo. If it was the difference between order-0 and order-1, it may be explained by the pcp allocator being bypassed.

On (28/02/08 20:48), Christoph Lameter didst pronounce:

Quoted 4 lines

This is the patchset that was posted two weeks ago modified according to the feedback that Pekka gave. I would like to put these patches into mm.

I haven't reviewed the patches properly but I put them through a quick test against 2.6.25-rc3 to see what the performnace was like and the superpage allocation success rates were like. Performance wise, it looked like

Loss to Gain
Kernbench Elapsed time -0.64% 0.32%
Kernbench Total time -0.61% 0.48%
Hackbench sockets-12 clients -2.95% 5.13%
Hackbench pipes-12 clients -16.95% 9.27%
TBench 4 clients -1.98% 8.2%
DBench 4 clients (ext2) -5.9% 7.99%

So, running with the high orders is not a clear-cut win to my eyes. What did you test to show that it was a general win justifying a high-order by default? From looking through, tbench seems to be the only obvious one to gain but the rest, it is not clear at all. I'll try give sysbench a spin later to see if it is clear-cut.

However, in *all* cases, superpage allocations were less successful and in some cases it was severely regressed (one machine went from 81% success rate to 36%). Sufficient statistics are not gathered to see why this happened in retrospect but my suspicion would be that high-order RECLAIMABLE and UNMOVABLE slub allocations routinely fall back to the less fragmented MOVABLE pageblocks with these patches - something that is normally a very rare event. This change in assumption hurts fragmentation avoidance and chances are the long-term behaviour of these patches is not great.

If this guess is correct, using a high-order size by default is a bad plan and it should only be set when it is known that the target workload benefits and superpage allocations are not a concern. Alternative, set high-order by default only for a limited number of caches that are RECLAIMABLE (or better yet ones we know can be directly reclaimed with the slub-defrag patches).

As it is, this is painful from a fragmentation perspective and the performance win is not clear-cut.

Thanks for looking at this, Mel. Could you try testing.. umm... slub_max_order=1? That's never going to get us more than one more object per slab, but if we can go from 1 per page to 1.5 per page, it might be worth it. Task structs are roughly in that size domain.