SLQB slab allocator (try 2)

Hi,

Since last time, fixed bugs pointed out by Hugh and Andi, cleaned up the code suggested by Ingo (haven't yet incorporated Ingo's last patch).

Should have fixed the crash reported by Yanmin (I was able to reproduce it on an ia64 system and fix it).

Significantly reduced static footprint of init arrays, thanks to Andi's suggestion.

Please consider for trial merge for linux-next.

Thanks,
Nick

--
Introducing the SLQB slab allocator.

SLQB takes code and ideas from all other slab allocators in the tree.

The primary method for keeping lists of free objects within the allocator is a singly-linked list, storing a pointer within the object memory itself (or a small additional space in the case of RCU destroyed slabs). This is like SLOB and SLUB, and opposed to SLAB, which uses arrays of objects, and metadata. This reduces memory consumption and makes smaller sized objects more realistic as there is less overhead.

Using lists rather than arrays can reduce the cacheline footprint. When moving objects around, SLQB can move a list of objects from one CPU to another by simply manipulating a head pointer, wheras SLAB needs to memcpy arrays. Some SLAB per-CPU arrays can be up to 1K in size, which is a lot of cachelines that can be touched during alloc/free. Newly freed objects tend to be cache hot, and newly allocated ones tend to soon be touched anyway, so often there is little cost to using metadata in the objects.

SLQB has a per-CPU LIFO freelist of objects like SLAB (but using lists rather than arrays). Freed objects are returned to this freelist if they belong to the node which our CPU belongs to. So objects allocated on one CPU can be added to the freelist of another CPU on the same node. When LIFO freelists need to be refilled or trimmed, SLQB takes or returns objects from a list of slabs.

SLQB has per-CPU lists of slabs (which use struct page as their metadata including list head for this list). Each slab contains a singly-linked list of objects that are free in that slab (free, and not on a LIFO freelist). Slabs are freed as soon as all their objects are freed, and only allocated when there are no slabs remaining. They are taken off this slab list when if there are no free objects left. So the slab lists always only contain "partial" slabs; those slabs which are not completely full and not completely empty. SLQB slabs can be manipulated with no locking unlike other allocators which tend to use per-node locks. As the number of threads per socket increases, this should help improve the scalability of slab operations.

Freeing objects to remote slab lists first batches up the objects on the freeing CPU, then moves them over at once to a list on the allocating CPU. The allocating CPU will then notice those objects and pull them onto the end of its freelist. This remote freeing scheme is designed to minimise the number of cross CPU cachelines touched, short of going to a "crossbar" arrangement like SLAB has. SLAB has "crossbars" of arrays of objects. That is, NR_CPUS*MAX_NUMNODES type arrays, which can become very bloated in huge systems (this could be hundreds of GBs for kmem caches for 4096 CPU, 1024 nodes systems).

SLQB also has similar freelist, slablist structures per-node, which are protected by a lock, and usable by any CPU in order to do node specific allocations. These allocations tend not to be too frequent (short lived allocations should be node local, long lived allocations should not be too frequent).

There is a good overview and illustration of the design here:

http://lwn.net/Articles/311502/

By using LIFO freelists like SLAB, SLQB tries to be very page-size agnostic. It tries very hard to use order-0 pages. This is good for both page allocator fragmentation, and slab fragmentation.

SLQB initialistaion code attempts to be as simple and un-clever as possible. There are no multiple phases where different things come up. There is no weird self bootstrapping stuff. It just statically allocates the structures required to create the slabs that allocate other slab structures.

SLQB uses much of the debugging infrastructure, and fine-grained sysfs statistics from SLUB. There is also a Documentation/vm/slqbinfo.c, derived from slabinfo.c, which can query the sysfs data.

 Documentation/vm/slqbinfo.c | 1054 +++++++++++++
 arch/x86/include/asm/page.h |    1
 include/linux/mm.h          |    4
 include/linux/rcu_types.h   |   18
 include/linux/rcupdate.h    |   11
 include/linux/slab.h        |   10
 include/linux/slqb_def.h    |  295 +++
 init/Kconfig                |    9
 lib/Kconfig.debug           |   20
 mm/Makefile                 |    1
 mm/slqb.c                   | 3562 ++++++++++++++++++++++++++++++++++++++++++++
 11 files changed, 4971 insertions(+), 14 deletions(-)

---

Index: linux-2.6/include/linux/rcupdate.h
===================================================================
@@ -33,6 +33,7 @@
 #ifndef __LINUX_RCUPDATE_H
 #define __LINUX_RCUPDATE_H
 
+#include <linux/rcu_types.h>
 #include <linux/cache.h>
 #include <linux/spinlock.h>
 #include <linux/threads.h>

@@ -42,16 +43,6 @@
 #include <linux/lockdep.h>
 #include <linux/completion.h>
 
-/**
- * struct rcu_head - callback structure for use with RCU
- * @next: next update requests in a list
- * @func: actual update function to call after the grace period.
- */
-struct rcu_head {
-	struct rcu_head *next;
-	void (*func)(struct rcu_head *head);
-};
-
 #if defined(CONFIG_CLASSIC_RCU)
 #include <linux/rcuclassic.h>
 #elif defined(CONFIG_TREE_RCU)

Index: linux-2.6/init/Kconfig
===================================================================
@@ -806,7 +806,7 @@ config SLUB_DEBUG
 
 choice
 	prompt "Choose SLAB allocator"
-	default SLUB
+	default SLQB
 	help
 	   This option allows to select a slab allocator.
 

@@ -827,6 +827,11 @@ config SLUB
 	   and has enhanced diagnostics. SLUB is the default choice for
 	   a slab allocator.
 
+config SLQB
+	bool "SLQB (Qeued allocator)"
+	help
+	  SLQB is a proposed new slab allocator.
+
 config SLOB
 	depends on EMBEDDED
 	bool "SLOB (Simple Allocator)"

@@ -868,7 +873,7 @@ config HAVE_GENERIC_DMA_COHERENT
 config SLABINFO
 	bool
 	depends on PROC_FS
-	depends on SLAB || SLUB_DEBUG
+	depends on SLAB || SLUB_DEBUG || SLQB
 	default y
 
 config RT_MUTEXES

Index: linux-2.6/lib/Kconfig.debug
===================================================================
@@ -298,6 +298,26 @@ config SLUB_STATS
 	  out which slabs are relevant to a particular load.
 	  Try running: slabinfo -DA
 
+config SLQB_DEBUG
+	default y
+	bool "Enable SLQB debugging support"
+	depends on SLQB
+
+config SLQB_DEBUG_ON
+	default n
+	bool "SLQB debugging on by default"
+	depends on SLQB_DEBUG
+
+config SLQB_SYSFS
+	bool "Create SYSFS entries for slab caches"
+	default n
+	depends on SLQB
+
+config SLQB_STATS
+	bool "Enable SLQB performance statistics"
+	default n
+	depends on SLQB_SYSFS
+
 config DEBUG_PREEMPT
 	bool "Debug preemptible kernel"
 	depends on DEBUG_KERNEL && PREEMPT && (TRACE_IRQFLAGS_SUPPORT || PPC64)

Index: linux-2.6/include/linux/slab.h
===================================================================

Index: linux-2.6/mm/Makefile
===================================================================

Index: linux-2.6/include/linux/rcu_types.h
===================================================================

Index: linux-2.6/arch/x86/include/asm/page.h
===================================================================

Index: linux-2.6/include/linux/mm.h
===================================================================

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Part of the diff is not shown here. View the raw message.

When applying the patch to 2.6.29-rc2, I got:
[ymzhang@lkp-h01 linux-2.6.29-rc2_slqb0123]$ patch -p1<../patch-slqb0123 patching file include/linux/rcupdate.h
patching file include/linux/slqb_def.h
patching file init/Kconfig
patching file lib/Kconfig.debug
patching file mm/slqb.c
patch: **** malformed patch at line 4042: Index: linux-2.6/include/linux/slab.h

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Hi Nick,

I merged a the one you resent privately as this one didn't apply at all. The code is in topic/slqb/core branch of slab.git and should appear in linux-next tomorrow.

Testing and especially performance testing is welcome. If any of the HPC people are reading this, please do give SLQB a good beating as Nick's plan is to replace both, SLAB and SLUB, with it in the long run. As Christoph has expressed concerns over latency issues of SLQB, I suppose it would be interesting to hear if it makes any difference to the real-time folks.

Pekka

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

I'll 'soon' take a stab at converting SLQB for -rt. Currently -rt is SLAB only.

Then again, anything that does allocation is per definition not bounded and not something we can have on latency critical paths -- so on that respect its not interesting.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Before someone pipes up, _yes_ I do know about RT allocators and such.

No we don't do that in-kernel, other than through reservation mechanisms like mempool -- and I'd rather extend that than try and get page reclaim bounded.

Yes, I also know about folks doing RT paging, and no, I'm not wanting to hear about that either ;-)

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Well there is the problem in SLAB and SLQB that they *continue* to do processing after an allocation. They defer queue cleaning. So your latency critical paths are interrupted by the deferred queue processing. SLAB has the awful habit of gradually pushing objects out of its queued (tried to approximate the loss of cpu cache hotness over time). So for awhile you get hit every 2 seconds with some free operations to the page allocator on each cpu. If you have a lot of cpus then this may become an ongoing operation. The slab pages end up in the page allocator queues which is then occasionally pushed back to the buddy lists. Another relatively high spike there.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

No they're not -- well, only if you let them that is, and then its your own fault.

Remember, -rt is about being able to preempt pretty much everything. If the userspace task has a higher priority than the timer interrupt, the timer interrupt just gets to wait.

Yes there is a very small hardirq window where the actual interrupt triggers, but all that that does is a wakeup and then its gone again.

Like Nick has been asking, can you give a solid test case that demonstrates this issue?

I'm thinking getting git of those cross-bar queues hugely reduces that problem.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

So you can have priority over kernel threads.... Sounds very dangerous.

Never used -rt. This is an issue seen in regular kernels.

Run a loop reading tsc and see the variances?

In HPC apps a series of processors have to sync repeatedly in order to complete operations. An event like cache cleaning can cause a disturbance in one processor that delays this sync in the system as a whole. And having it run at offsets separately on all processor causes the disturbance to happen on one processor after another. In extreme cases all syncs are delayed. We have seen this effect have a major delay on HPC app performance.

Note that SLAB scans through all slab caches in the system and expires queues that are active. The more slab caches there are and the more data is in queues the longer the process takes.

The cross-bar queues are a significant problem because they mean operation on objects that are relatively far away. So the time spend in cache cleaning increases significantly. But as far as I can see SLQB also has cross-bar queues like SLAB. SLUB does all necessary actions during the actual allocation or free so there is no need to run cache cleaning.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Groan.

Now we are starting to get somewhere slightly useful. Can we have more details about this workload please? Was there a test program coded up to run the same sequence of MPI operations? Or can they at least be described?

And the larger the number of nodes and CPUs, because SLAB can have so many queues. This is not an issue with SLQB, so I don't think it will be subject to the same magnitude of problem on your large machines.

Periodic cleaning in SLAB was never shown to be a problem with large HPC clusters in the past, so that points to SGI's problem as being due to explosion of queues in big machines rather than the whole concept of periodic cleaning.

And we have lots of periodic things going on. Periodic journal flushing, periodic dirty watermark checking, periodic timers, multiprocessor CPU scheduler balancing etc etc. So no, I totally reject the assertion that periodic slab cleaning is a showstopper. Without actual numbers or test cases, I don't need to hear any more assertions in this vein.

(But note, numbers and/or test cases etc would be very very welcome because I would like to tune SLQB performance on HPC as much as possible and as I have already said, there are ways we can improve or mitigate periodic trimming overheads).

Well it doesn't.

And SLUB can actually leave free pages lying around that never get cleaned up because of this. As I said, I have seen SLQB use less memory than SLUB in some situations I assume because of this (although now that I think about it, perhaps it was due to increased internal fragmentation in bigger pages).

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

The HPC folks care about a few different workloads but speccpu is one that shows up. I was in the position to run tests because I had put together the test harness for a paper I spent the last month writing. This mail shows a comparison between slab, slub and slqb for speccpu2006 running a single thread and sysbench ranging clients from 1 to 4*num_online_cpus() (16 in both cases). Additional tests were not run because just these two take one day per kernel to complete. Results are ratios to the SLAB figures and based on an x86-64 and ppc64 machine.

X86-64 Test machine
CPU AMD Phenom 9950 Quad-Core
CPU Frequency 1.3GHz
Physical CPUs 1 (4 cores)
L1 Cache 64K Data, 64K Instruction per core
L2 Cache 512K Unified per core
L3 Cache 2048K Unified Shared per chip
Main Memory 8 GB
Mainboard Gigabyte GA-MA78GM-S2H
Machine Model Custom built from parts

SPEC CPU 2006
-------------
Integer tests

SPEC test       slab         slub       slqb
400.perlbench   1.0000     1.0016     1.0064
401.bzip2       1.0000     0.9804     1.0011
403.gcc         1.0000     1.0023     0.9965
429.mcf         1.0000     1.0022     0.9963
445.gobmk       1.0000     0.9944     0.9986
456.hmmer       1.0000     0.9792     0.9701
458.sjeng       1.0000     0.9989     1.0133
462.libquantum  1.0000     0.9905     0.9981
464.h264ref     1.0000     0.9877     1.0058
471.omnetpp     1.0000     0.9893     1.0993
473.astar       1.0000     0.9542     0.9596
483.xalancbmk   1.0000     0.9547     0.9982

---------------
specint geomean 1.0000 0.9862 1.0031

Floating Point Tests

SPEC test       slab         slub       slqb
410.bwaves      1.0000     0.9939     1.0005
416.gamess      1.0000     1.0040     0.9984
433.milc        1.0000     0.9865     0.9865
434.zeusmp      1.0000     0.9810     0.9879
435.gromacs     1.0000     0.9854     1.0125
436.cactusADM   1.0000     1.0467     1.0294
437.leslie3d    1.0000     0.9846     0.9963
444.namd        1.0000     1.0000     1.0000
447.dealII      1.0000     0.9913     0.9957
450.soplex      1.0000     0.9940     1.0015
453.povray      1.0000     0.9904     1.0197
454.calculix    1.0000     0.9937     1.0000
459.GemsFDTD    1.0000     1.0061     1.0000
465.tonto       1.0000     0.9979     0.9989
470.lbm         1.0000     1.0099     1.0212
481.wrf         1.0000     1.0000     1.0045
482.sphinx3     1.0000     1.0047     1.0068

---------------
specfp geomean 1.0000 0.9981 1.0035

Sysbench - Postgres
-------------------
Client slab slub slqb

     1          1.0000     0.9484     0.9804
     2          1.0000     1.0069     0.9994
     3          1.0000     1.0064     0.9994
     4          1.0000     0.9900     0.9904
     5          1.0000     1.0023     0.9869
     6          1.0000     1.0139     1.0069
     7          1.0000     0.9973     0.9991
     8          1.0000     1.0206     1.0197
     9          1.0000     0.9884     0.9817
    10          1.0000     0.9980     1.0135
    11          1.0000     0.9959     1.0164
    12          1.0000     0.9978     0.9953
    13          1.0000     1.0024     0.9942
    14          1.0000     0.9975     0.9808
    15          1.0000     0.9914     0.9933
    16          1.0000     0.9767     0.9726

--------------
Geometric mean 1.0000 0.9957 0.9955

On this particular x86-64, slab is on average faster for sysbench but by a very small margin, less then 0.5%. I wasn't doing multiple runs for each client number to see if this is within the noise but generally these figures are quite stable. SPEC CPU is more interesting. Both SLUB and SLQB regress on a number of the benchmarks although on average SLQB is very marginally faster than SLAB (approx 0.3% faster) where SLUB is between around 1% slower. Both SLUB and SLQB show big regressions on some tests:
hmmer, astar. omnetpp is also interesting in that SLUB regresses a little and SLQB gains considerably. This is likely due to luck in cache placement.

Overall, while the regressions where they exist are troublesome, they are also small and I strongly suspect there are far greater variances between kernel releases due to changes other than the allocators. SLQB is the winner, but by a minimal margin.

PPC64 Test Machine
CPU PPC970MP, altivec supported
CPU Frequency 2.5GHz
Physical CPUs 2 x dual core (4 cores in all)
L1 Cache 32K Data, 64K Instruction per core
L2 Cache 1024K Unified per core
L3 Cache N/a
Main Memory 10GB
Mainboard Specific to the machine model

SPEC CPU 2006
-------------
Integer tests

SPEC test       slab         slub       slqb
400.perlbench   1.0000     1.0497     1.0497
401.bzip2       1.0000     1.0496     1.0489
403.gcc         1.0000     1.0509     1.0509
429.mcf         1.0000     1.0554     1.0549
445.gobmk       1.0000     1.0535     1.0556
456.hmmer       1.0000     1.0651     1.0566
458.sjeng       1.0000     1.0612     1.0564
462.libquantum  1.0000     1.0389     1.0396
464.h264ref     1.0000     1.0517     1.0503
471.omnetpp     1.0000     1.0555     1.0574
473.astar       1.0000     1.0508     1.0521
483.xalancbmk   1.0000     1.0594     1.0584

---------------
specint geomean 1.0000 1.0534 1.0525

Floating Point Tests

SPEC test       slab         slub       slqb
410.bwaves      1.0000     1.0381     1.0367
416.gamess      1.0000     1.0550     1.0550
433.milc        1.0000     1.0464     1.0450
434.zeusmp      1.0000     1.0510     1.0528
435.gromacs     1.0000     1.0461     1.0445
436.cactusADM   1.0000     1.0457     1.0450
437.leslie3d    1.0000     1.0437     1.0428
444.namd        1.0000     1.0482     1.0496
447.dealII      1.0000     1.0505     1.0505
450.soplex      1.0000     1.0522     1.0499
453.povray      1.0000     1.0513     1.0534
454.calculix    1.0000     1.0374     1.0357
459.GemsFDTD    1.0000     1.0465     1.0465
465.tonto       1.0000     1.0488     1.0456
470.lbm         1.0000     1.0438     1.0452
481.wrf         1.0000     1.0423     1.0429
482.sphinx3     1.0000     1.0464     1.0479

---------------
specfp geomean 1.0000 1.0467 1.0464

Sysbench - Postgres
-------------------
Client slab slub slqb

     1          1.0000     1.0153     1.0051
     2          1.0000     1.0273     1.0269
     3          1.0000     1.0299     1.0234
     4          1.0000     1.0159     1.0146
     5          1.0000     1.0232     1.0264
     6          1.0000     1.0238     1.0088
     7          1.0000     1.0240     1.0076
     8          1.0000     1.0134     1.0024
     9          1.0000     1.0154     1.0077
    10          1.0000     1.0126     1.0009
    11          1.0000     1.0100     0.9933
    12          1.0000     1.0112     0.9993
    13          1.0000     1.0131     1.0035
    14          1.0000     1.0237     1.0071
    15          1.0000     1.0098     0.9997
    16          1.0000     1.0110     0.9994
Geometric mean  1.0000     1.0175     1.0078

Unlike x86-64, ppc64 sees a consistent gain with with SLUB or SLQB and the difference between SLUB and SLQB negligible on average with speccpu. However, it is very noticeable sysbench where SLUB is generally a win in the 1% range over SLQB and SLQB regressed very marginally in a few instances. The three benchmarks showing odd behaviour on x86-64, hmmer aster and omnetpp, do not show similar weirdness on ppc64.

Overall on ppc64, SLUB is the winner by a clearer margin.

Summary
-------
The decision on whether to use SLUB, SLAB or SLQB for either speccpu 2006 and sysbench is not very clearcut. The win on ppc64 would imply SLUB is the way to go but it is also clear that different architectures will produce different results which needs to be taken into account when trying to reproduce figures from other people. I stronly suspect variations of the same architecture will also show different results. Things like omnetpp on x86-64 imply that how cache is used is a factor but it's down to luck how what the result will be.

Downsides
---------
The SPEC CPU tests were not parallelised so there is no indication as to which allocator might scale better to the number of CPUs. The test machines were not NUMA so there is also no indication on which might be better there. There wasn't a means of measuring allocator jitter. Queue cleaning means that some allocations might stall, something that SLUB is expected to be immune to, doesn't have a clear way of measuring but something other allocators could potentially "cheat" on by postponing cleaning to a time when performance is not being measured. I wasn't tracking to see which consumed the least memory so we don't know which is more memory efficient either.

The OLTP workload results could indicate a downside with using sysbench although it could also be hardware. The reports from the Intel guys have been pretty clear-cut that SLUB is a loser but sysbench-postgres on these test machines at least do not agree. Of course their results are perfectly valid but the discrepency needs to be explained or there will be a disconnect between developers and the performance people. Something important is missing that means sysbench-postgres *may* not be a reliable indicator of TPC-C performance. It could easily be down to the hardware as their tests are on a mega-large machine with oodles of disks and probably NUMA where the test machine used for this is a lot less respectable.

Mel Gorman
Part-time Phd Student                          Linux Technology Center
University of Limerick                         IBM Dublin Software Lab

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Hi Mel,

This is very nice, thanks for testing. SLQB and SLUB are quite similar in a lot of cases, which indeed could be explained by cacheline placement (both of these can allocate down to much smaller sizes, and both of them also put metadata directly in free object memory rather than external locations).

But it will be interesting to try looking at some of the tests where SLQB has larger regressions, so that might give me something to go on if I can lay my hands on speccpu2006...

I'd be interested to see how slub performs if booted with slub_min_objects=1 (which should give similar order pages to SLAB and SLQB).

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Sure. It's been on my TODO list for long enough :). I should have been clear that the ratios are performance improvements based on wall time. A result of 0.9862 implies a performance regression of 1.38% in comparison to SLAB. 1.0031 implies a performance gain of 0.31% etc.

Indeed. I know from other tests that poor cacheline placement can crucify performance. My current understanding is we don't notice as data and metadata are effectively using random cache lines.

I can generate profile runs although it'll take 3 days to gather it all together unless I target specific tests (the worst ones to start with obviously). The suite has a handy feature called monitor hooks that allows a pre and post script to run for each test which I use it to start/stop oprofile and gather one report per benchmark. I didn't use it for this run as profiling affects the outcome (7-9% overhead).

I do have detailed profile data available for sysbench, both per thread run and the entire run but with the instruction-level included, it's a lot of data to upload. If you still want it, I'll start it going and it'll get up there eventually.

I'll do this before profiling as only one run is required and should only take a day.

Making spec actually build is tricky so I've included a sample config for x86-64 below that uses gcc and the monitor hooks in case someone else is in the position to repeat the results.

===== Begin sample spec config file =====
# Autogenerated by generate-speccpu.sh

## Base configuration

ignore_errors      = no
tune               = base
ext                = x86_64-m64-gcc42
output_format      = asc, pdf, Screen
reportable         = 1
teeout             = yes
teerunout          = yes
hw_avail           = September 2008
license_num        = 
test_sponsor       = 
prepared_by        = Mel Gorman
tester             = Mel Gorman
test_date          = Dec 2008

## Compiler

CC                 = gcc-4.2
CXX                = g++-4.2
FC                 = gfortran-4.2

## HW config

hw_model           = Gigabyte Technology Co., Ltd. GA-MA78GM-S2H
hw_cpu_name        = AMD Phenom(tm) 9950 Quad-Core Processor
hw_cpu_char        = 
hw_cpu_mhz         = 1300.000
hw_fpu             = Integrated
hw_nchips          = 1
hw_ncores          = 4
hw_ncoresperchip   = 4
hw_nthreadspercore = 1
hw_ncpuorder       = 
hw_pcache          = L1 64K Data, 64K Instruction per core
hw_scache          = L2 512K Unified per core
hw_tcache          = L3 2048K Unified Shared per chip
hw_ocache          = 
hw_memory          = 4594MB
hw_disk            = SATA WD5000AAKS-00A7B0
hw_vendor          = Komplett.ie

## SW config

sw_os              = Debian Lenny Beta for x86_64
sw_file            = ext3
sw_state           = Runlevel [2]
sw_compiler        = gcc, g++ & gfortran 4.2 for x86_64
sw_avail           = Dec 2008
sw_other           = None
sw_auto_parallel   = No
sw_base_ptrsize    = 64-bit
sw_peak_ptrsize    = Not Applicable

## Monitor hooks
monitor_pre_bench = /home/mel/git-public/vmregress/bin/oprofile_start.sh --event timer --event dtlb_miss; echo iter >> /tmp/OPiter.${lognum}.${size_class}.${benchmark}
monitor_post_bench = opcontrol --stop ; /home/mel/git-public/vmregress/bin/oprofile_report.sh > dirname ${logname}/OP.${lognum}.${size_class}.itercat /tmp/OPiter.${lognum}.${size_class}.${benchmark} | wc -l.${benchmark}.txt

## Optimisation

makeflags          = -j4
COPTIMIZE          = -O2 -m64
CXXOPTIMIZE        = -O2 -m64
FOPTIMIZE          = -O2 -m64

notes0100= C base flags: $[COPTIMIZE]
notes0110= C++ base flags: $[CXXOPTIMIZE]
notes0120= Fortran base flags: $[FOPTIMIZE]

## Portability flags - all
default=base=default=default:
notes35 = PORTABILITY=-DSPEC_CPU_LP64 is applied to all benchmarks
PORTABILITY = -DSPEC_CPU_LP64

## Portability flags - int
400.perlbench=default=default=default:
CPORTABILITY = -DSPEC_CPU_LINUX_X64
notes35 = 400.perlbench: -DSPEC_CPU_LINUX_X64

462.libquantum=default=default=default:
CPORTABILITY = -DSPEC_CPU_LINUX
notes60 = 462.libquantum: -DSPEC_CPU_LINUX

483.xalancbmk=default=default=default:
CXXPORTABILITY = -DSPEC_CPU_LINUX

## Portability flags - flt
481.wrf=default=default=default:
CPORTABILITY = -DSPEC_CPU_CASE_FLAG -DSPEC_CPU_LINUX

__MD5__

Mel Gorman
Part-time Phd Student                          Linux Technology Center
University of Limerick                         IBM Dublin Software Lab

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

/me slaps self

For SPEC, it's performance improvements based on wall time as measured by the suite. For sysbench, it's performance improvements based on operations per second.

Mel Gorman
Part-time Phd Student                          Linux Technology Center
University of Limerick                         IBM Dublin Software Lab

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

It couldn't hurt, but it's usually tricky to read anything out of these from CPU cycle profiles. Especially if they are due to cache or tlb effects (which tend to just get spread out all over the profile).

slabinfo (for SLUB) and slqbinfo (for SLQB) activity data could be interesting (invoke with -AD).

Thanks. I don't know if we have a copy of spec 2006 I can use, but I'll ask around.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Indeed. To date, I've used them for comparing relative counts of things like TLB and cache misses on the basis "relatively more misses running test X is bad" or working out things like tlb-misses-per-instructions but it's a bit vague. We might notice if one of the allocators is being particularly cache unfriendly due to a spike in cache misses.

Ok, I butchered Ingo's proc monitoring script to gather /proc/slabinfo, slabinfo -AD and slqbinfo -AD before and after each speccpu subtest. The tests with profiling just started but it will take a few days to complete and thats assuming I made no mistakes in the automation. I'll be at FOSDEM from Friday till Monday so may not be able to collect the results until Monday.

In the meantime, here are the results I have with slub configured to use small orders.

X86-64 Test machine

        CPU             AMD Phenom 9950 Quad-Core
        CPU Frequency   1.3GHz
        Physical CPUs   1 (4 cores)
        L1 Cache        64K Data, 64K Instruction per core
        L2 Cache        512K Unified per core
        L3 Cache        2048K Unified Shared per chip
        Main Memory     8 GB
        Mainboard       Gigabyte GA-MA78GM-S2H
        Machine Model   Custom built from parts

SPEC CPU 2006
-------------
Integer tests

SPEC test       slab                 slub  slub-minorder           slqb
400.perlbench   1.0000             1.0016         0.9921         1.0064
401.bzip2       1.0000             0.9804         0.9858         1.0011
403.gcc         1.0000             1.0023         0.9977         0.9965
429.mcf         1.0000             1.0022         0.9847         0.9963
445.gobmk       1.0000             0.9944         0.9958         0.9986
456.hmmer       1.0000             0.9792         0.9874         0.9701
458.sjeng       1.0000             0.9989         1.0144         1.0133
462.libquantum  1.0000             0.9905         0.9943         0.9981
464.h264ref     1.0000             0.9877         0.9926         1.0058
471.omnetpp     1.0000             0.9893         1.0896         1.0993
473.astar       1.0000             0.9542         0.9930         0.9596
483.xalancbmk   1.0000             0.9547         0.9928         0.9982

---------------
specint geomean 1.0000 0.9862 1.0013 1.0031

Floating Point Tests

SPEC test       slab                 slub  slub-minorder           slqb
410.bwaves      1.0000             0.9939         1.0000         1.0005
416.gamess      1.0000             1.0040         1.0032         0.9984
433.milc        1.0000             0.9865         0.9986         0.9865
434.zeusmp      1.0000             0.9810         0.9980         0.9879
435.gromacs     1.0000             0.9854         1.0100         1.0125
436.cactusADM   1.0000             1.0467         0.9904         1.0294
437.leslie3d    1.0000             0.9846         0.9970         0.9963
444.namd        1.0000             1.0000         0.9986         1.0000
447.dealII      1.0000             0.9913         0.9957         0.9957
450.soplex      1.0000             0.9940         0.9955         1.0015
453.povray      1.0000             0.9904         1.0097         1.0197
454.calculix    1.0000             0.9937         0.9975         1.0000
459.GemsFDTD    1.0000             1.0061         0.9902         1.0000
465.tonto       1.0000             0.9979         1.0000         0.9989
470.lbm         1.0000             1.0099         0.9924         1.0212
481.wrf         1.0000             1.0000         1.0045         1.0045
482.sphinx3     1.0000             1.0047         1.0000         1.0068

---------------
specfp geomean 1.0000 0.9981 0.9989 1.0035

Sysbench-Postgres
-----------------
Client slab slub-default slub-minorder slqb

     1         1.0000        0.9484         0.9699          0.9804
     2         1.0000        1.0069         1.0036          0.9994
     3         1.0000        1.0064         1.0080          0.9994
     4         1.0000        0.9900         1.0049          0.9904
     5         1.0000        1.0023         1.0144          0.9869
     6         1.0000        1.0139         1.0215          1.0069
     7         1.0000        0.9973         0.9966          0.9991
     8         1.0000        1.0206         1.0223          1.0197
     9         1.0000        0.9884         1.0167          0.9817
    10         1.0000        0.9980         0.9842          1.0135
    11         1.0000        0.9959         1.0036          1.0164
    12         1.0000        0.9978         1.0032          0.9953
    13         1.0000        1.0024         1.0022          0.9942
    14         1.0000        0.9975         1.0064          0.9808
    15         1.0000        0.9914         0.9949          0.9933
    16         1.0000        0.9767         0.9692          0.9726
Geo. mean      1.0000        0.9957         1.0012          0.9955

PPC64 Test Machine
CPU PPC970MP, altivec supported
CPU Frequency 2.5GHz
Physical CPUs 2 x dual core (4 cores in all)
L1 Cache 32K Data, 64K Instruction per core
L2 Cache 1024K Unified per core
L3 Cache N/a
Main Memory 10GB
Mainboard Specific to the machine model

SPEC CPU 2006
-------------
Integer tests

SPEC test       slab                 slub  slub-minorder           slqb
400.perlbench   1.0000             1.0497         1.0515         1.0497
401.bzip2       1.0000             1.0496         1.0496         1.0489
403.gcc         1.0000             1.0509         1.0509         1.0509
429.mcf         1.0000             1.0554         1.0549         1.0549
445.gobmk       1.0000             1.0535         1.0545         1.0556
456.hmmer       1.0000             1.0651         1.0636         1.0566
458.sjeng       1.0000             1.0612         1.0612         1.0564
462.libquantum  1.0000             1.0389         1.0403         1.0396
464.h264ref     1.0000             1.0517         1.0496         1.0503
471.omnetpp     1.0000             1.0555         1.0574         1.0574
473.astar       1.0000             1.0508         1.0514         1.0521
483.xalancbmk   1.0000             1.0594         1.0584         1.0584

---------------
specint geomean 1.0000 1.0534 1.0536 1.0525

Floating Point Tests

SPEC test       slab                 slub  slub-minorder           slqb
410.bwaves      1.0000             1.0381         1.0381         1.0367
416.gamess      1.0000             1.0550         1.0539         1.0550
433.milc        1.0000             1.0464         1.0457         1.0450
434.zeusmp      1.0000             1.0510         1.0482         1.0528
435.gromacs     1.0000             1.0461         1.0437         1.0445
436.cactusADM   1.0000             1.0457         1.0463         1.0450
437.leslie3d    1.0000             1.0437         1.0437         1.0428
444.namd        1.0000             1.0482         1.0482         1.0496
447.dealII      1.0000             1.0505         1.0495         1.0505
450.soplex      1.0000             1.0522         1.0511         1.0499
453.povray      1.0000             1.0513         1.0534         1.0534
454.calculix    1.0000             1.0374         1.0370         1.0357
459.GemsFDTD    1.0000             1.0465         1.0465         1.0465
465.tonto       1.0000             1.0488         1.0494         1.0456
470.lbm         1.0000             1.0438         1.0438         1.0452
481.wrf         1.0000             1.0423         1.0423         1.0429
482.sphinx3     1.0000             1.0464         1.0479         1.0479

---------------
specfp geomean 1.0000 1.0467 1.0464 1.0464

Sysbench-Postgres
-----------------
Client slab slub-default slub-minorder slqb

     1         1.0000        1.0153         1.0179          1.0051
     2         1.0000        1.0273         1.0181          1.0269
     3         1.0000        1.0299         1.0195          1.0234
     4         1.0000        1.0159         1.0130          1.0146
     5         1.0000        1.0232         1.0192          1.0264
     6         1.0000        1.0238         1.0142          1.0088
     7         1.0000        1.0240         1.0063          1.0076
     8         1.0000        1.0134         0.9842          1.0024
     9         1.0000        1.0154         1.0152          1.0077
    10         1.0000        1.0126         1.0018          1.0009
    11         1.0000        1.0100         0.9971          0.9933
    12         1.0000        1.0112         0.9985          0.9993
    13         1.0000        1.0131         1.0060          1.0035
    14         1.0000        1.0237         1.0074          1.0071
    15         1.0000        1.0098         0.9997          0.9997
    16         1.0000        1.0110         0.9899          0.9994
Geo. mean      1.0000        1.0175         1.0067          1.0078

The order SLUB uses does not make much of a difference to SPEC CPU on either test machine or sysbench on x86-64. Howeer, on the ppc64 machine, the performance advantage SLUB has over SLAB appears to be eliminated if high-order pages are not used. I think I might run SLUB again incase the higher average performance was a co-incidence due to lucky cache layout. Otherwise, Christoph can probably put together a plausible theory on this result faster than I can.

On the TLB front, it is perfectly possible that the workloads on x86-64 are not allocator or memory intensive enough to take advantage of fewer calls to the page allocator or potentially reduced TLB pressure. As the kernel portion of the address space already uses huge pages slab objects may have to occupy a very large percentage of memory before TLB pressure became an issue. The L1 TLBs on both test machines are fully associative making testing reduced TLB pressure practically impossible. For bonus points, 1G pages are being used on the x86-64 so I have nowhere near enough memory to put that under TLB pressure.

Measuring reduced metadata overhead is more plausible.

Mel Gorman
Part-time Phd Student                          Linux Technology Center
University of Limerick                         IBM Dublin Software Lab

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Very true. Total counts of TLB and cache misses could show some insight.

It's interesting, thanks. It's a good result for SLQB I guess. 1% is fairly large here (if it is statistically significant), but I don't think the drawbacks of using higher order pages warrant changing anything by default in SLQB. It does encourage me to add a boot or runtime parameter, though (even if just for testing purposes).

TLB pressure... I would be interested in. I'm not exactly sold on the idea that higher order allocations will give a significant TLB improvement. Although for benchmark runs, maybe it is more likely (ie. if memory hasn't been too fragmented).

Suppose you have a million slab objects scattered all over memory, the fact you might have them clumped into 64K regions rather than 4K regions... is it going to be significant? How many access patterns are likely to soon touch exactly those objects that are in the same page?

Sure it is possible to come up with a scenario where it does help. But also others where it will not.

OTOH, if it is a win on ppc but not x86-64, then that may point to TLB...

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Agreed. I'm collecting just cache misses in this run. Due to limitations on the ppc970 PMU, I can't collect TLB and cache at the same time. I can't remember if it can or not on the x86-64 but I went with the lowest common denominator in any case.

I believe it is. I don't recall the figures deviating much for sysbench but I have to alter the scripts to do multiple runs just in case.

Based on this test-machine, it's not justified by default but as the tests are not allocator intensive it doesn't say much. tbench (or netperf or anything that is more slab intensive) might reveal something but it'll be Monday at the earliest before I can find out.

Currently, I suspect the machine has to be running a very long time and the memory footprint used by SLAB has to be significant before a large enough number of TLB entries are being used. A side-effect of anti-fragmentation is that kernel allocations get grouped into hugepages as much as possible without using high-order pages. This makes it even harder to cause TLB pressure within the kernel (a good thing in general).

I doubt it's significant from a TLB perspective. Anti-frag will be clumping the the 4K pages together in 2MB (on x86-64) and 16MB (on ppc64) already. It would actually be pretty tricky to form an allocation pattern from userspace that would force use of multiple huge pages. While it is possible thh situation does occur, I don't think a realistic test-case can be put together that demonstrates it.

What 64K regions will do is reduce the amount of management data needed by slab and reduce the number of calls to the page allocator. This is likely to be much more significant in general than TLBs.

I'm not sure what it is, but I'm not convinced right now that TLB could make a 1% difference. Maybe the cache miss figures will show something up and if not, and there is nothign obvious in the profiles, I'll rerun with TLB profiling and see what pops up.

Mel Gorman
Part-time Phd Student                          Linux Technology Center
University of Limerick                         IBM Dublin Software Lab

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Slightly later than hoped for, but here are the results of the profile run between the different slab allocators. It also includes information on the performance on SLUB with the allocator pass-thru logic reverted by commit
http://git.kernel.org/?p=linux/kernel/git/penberg/slab-2.6.git;a=commitdiff;h=97a4871761e735b6f1acd3bc7c3bac30dae3eab9

Hi Mel,

So how big does an improvement/regression have to be not to be considered within noise? I mean, I randomly picked one of the results ("x86-64 speccpu integer tests") and ran it through my "summarize" script and got the following results:

min max mean std_dev
slub 0.96 1.09 1.01 0.04
slub-min 0.95 1.10 1.00 0.04
slub-rvrt 0.90 1.08 0.99 0.05
slqb 0.96 1.07 1.00 0.04

Apart from slub-rvrt (which seems to be regressing, interesting) all the allocators seem to perform equally well. Hmm?

Btw, Yanmin, do you have access to the tests Mel is running (especially the ones where slub-rvrt seems to do worse)? Can you see this kind of regression? The results make we wonder whether we should avoid reverting all of the page allocator pass-through and just add a kmalloc cache for 8K allocations. Or not address the netperf regression at all. Double-hmm.

Pekka

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Well, it doesn't make a whole pile of sense to get the average of these ratios or the deviation between them. Each of the tests behave very differently. I'd consider anything over 0.5% significant but I also have to admit I wasn't doing multiple runs this time due to the length of time it takes. In a previous test, I ran them 3 times each and didn't spot large deviations.

For this stuff, they are reasonably close but I don't believe thye are allocator intensive either. SPEC CPU was brought up as a workload HPC people would care about. Bear in mind it's also not testing NUMA or CPU scalability really well. It's one data-point. netperf is a much more allocator intensive workload.

Mel Gorman
Part-time Phd Student                          Linux Technology Center
University of Limerick                         IBM Dublin Software Lab

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Uhm, yes. I need to learn to read one of these days.

Pekka

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

I wonder if different compilation of kernel might cause different cache alignment which has much impact on small result difference.

If a workload isn't slab-allocation intensive, perhaps the impact caused by different compilation is a little bigger.

As it takes a long time (more than 20 hours) to run cpu2006, I run cpu2000 instead of cpu2006. Now, we are trying to integrate cpu2006 into testing infrastructure. i>>?Let me check it firstly.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Going to 8k for the limit beyond we pass through to the page allocator may be the simplest and best solution. Someone please work on the page allocator...

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Hi Christoph,

Yeah. Something like this totally untested patch, perhaps?

Pekka

index 2f5c16b..e93cb3d 100644
@@ -201,6 +201,13 @@ static __always_inline struct kmem_cache *kmalloc_slab(size_t size)
 #define SLUB_DMA (__force gfp_t)0
 #endif
 
+/*
+ * The maximum allocation size that will be satisfied by the slab allocator for
+ * kmalloc(). Requests that exceed this limit are passed directly to the page
+ * allocator.
+ */
+#define SLAB_LIMIT (8 * 1024)
+
 void *kmem_cache_alloc(struct kmem_cache *, gfp_t);
 void *__kmalloc(size_t size, gfp_t flags);
 

@@ -212,7 +219,7 @@ static __always_inline void *kmalloc_large(size_t size, gfp_t flags)
 static __always_inline void *kmalloc(size_t size, gfp_t flags)
 {
 	if (__builtin_constant_p(size)) {
-		if (size > PAGE_SIZE)
+		if (size > SLAB_LIMIT)
 			return kmalloc_large(size, flags);
 
 		if (!(flags & SLUB_DMA)) {

@@ -234,7 +241,7 @@ void *kmem_cache_alloc_node(struct kmem_cache *, gfp_t flags, int node);
 static __always_inline void *kmalloc_node(size_t size, gfp_t flags, int node)
 {
 	if (__builtin_constant_p(size) &&
-		size <= PAGE_SIZE && !(flags & SLUB_DMA)) {
+		size <= SLAB_LIMIT && !(flags & SLUB_DMA)) {
 			struct kmem_cache *s = kmalloc_slab(size);
 
 		if (!s)

index 0280eee..a324188 100644
@@ -2658,7 +2658,7 @@ void *__kmalloc(size_t size, gfp_t flags)
 {
 	struct kmem_cache *s;
 
-	if (unlikely(size > PAGE_SIZE))
+	if (unlikely(size > SLAB_LIMIT))
 		return kmalloc_large(size, flags);
 
 	s = get_slab(size, flags);

@@ -2686,7 +2686,7 @@ void *__kmalloc_node(size_t size, gfp_t flags, int node)
 {
 	struct kmem_cache *s;
 
-	if (unlikely(size > PAGE_SIZE))
+	if (unlikely(size > SLAB_LIMIT))
 		return kmalloc_large_node(size, flags, node);
 
 	s = get_slab(size, flags);

@@ -3223,7 +3223,7 @@ void *__kmalloc_track_caller(size_t size, gfp_t gfpflags, unsigned long caller)
 {
 	struct kmem_cache *s;
 
-	if (unlikely(size > PAGE_SIZE))
+	if (unlikely(size > SLAB_LIMIT))
 		return kmalloc_large(size, gfpflags);
 
 	s = get_slab(size, gfpflags);

@@ -3239,7 +3239,7 @@ void *__kmalloc_node_track_caller(size_t size, gfp_t gfpflags,
 {
 	struct kmem_cache *s;
 
-	if (unlikely(size > PAGE_SIZE))
+	if (unlikely(size > SLAB_LIMIT))
 		return kmalloc_large_node(size, gfpflags, node);
 
 	s = get_slab(size, gfpflags);

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Well yes you missed two locations (kmalloc_caches array has to be redimensioned) and I also was writing the same patch...

Here is mine:

Subject: SLUB: Do not pass 8k objects through to the page allocator

Increase the maximum object size in SLUB so that 8k objects are not passed through to the page allocator anymore. The network stack uses 8k objects for performance critical operations.

Index: linux-2.6/include/linux/slub_def.h
===================================================================
@@ -121,10 +121,21 @@
 #define KMALLOC_SHIFT_LOW ilog2(KMALLOC_MIN_SIZE)
 
 /*
+ * Maximum kmalloc object size handled by SLUB. Larger object allocations
+ * are passed through to the page allocator. The page allocator "fastpath"
+ * is relatively slow so we need this value sufficiently high so that
+ * performance critical objects are allocated through the SLUB fastpath.
+ *
+ * This should be dropped to PAGE_SIZE / 2 once the page allocator
+ * "fastpath" becomes competitive with the slab allocator fastpaths.
+ */
+#define SLUB_MAX_SIZE (2 * PAGE_SIZE)
+
+/*
  * We keep the general caches in an array of slab caches that are used for
  * 2^x bytes of allocations.
  */
-extern struct kmem_cache kmalloc_caches[PAGE_SHIFT + 1];
+extern struct kmem_cache kmalloc_caches[PAGE_SHIFT + 2];
 
 /*
  * Sorry that the following has to be that ugly but some versions of GCC

@@ -212,7 +223,7 @@
 static __always_inline void *kmalloc(size_t size, gfp_t flags)
 {
 	if (__builtin_constant_p(size)) {
-		if (size > PAGE_SIZE)
+		if (size > SLUB_MAX_SIZE)
 			return kmalloc_large(size, flags);
 
 		if (!(flags & SLUB_DMA)) {

@@ -234,7 +245,7 @@
 static __always_inline void *kmalloc_node(size_t size, gfp_t flags, int node)
 {
 	if (__builtin_constant_p(size) &&
-		size <= PAGE_SIZE && !(flags & SLUB_DMA)) {
+		size <= SLUB_MAX_SIZE && !(flags & SLUB_DMA)) {
 			struct kmem_cache *s = kmalloc_slab(size);
 
 		if (!s)

Index: linux-2.6/mm/slub.c
===================================================================
@@ -2475,7 +2475,7 @@
  *		Kmalloc subsystem
  *******************************************************************/
 
-struct kmem_cache kmalloc_caches[PAGE_SHIFT + 1] __cacheline_aligned;
+struct kmem_cache kmalloc_caches[PAGE_SHIFT + 2] __cacheline_aligned;
 EXPORT_SYMBOL(kmalloc_caches);
 
 static int __init setup_slub_min_order(char *str)

@@ -2658,7 +2658,7 @@
 {
 	struct kmem_cache *s;
 
-	if (unlikely(size > PAGE_SIZE))
+	if (unlikely(size > SLUB_MAX_SIZE))
 		return kmalloc_large(size, flags);
 
 	s = get_slab(size, flags);

@@ -2686,7 +2686,7 @@
 {
 	struct kmem_cache *s;
 
-	if (unlikely(size > PAGE_SIZE))
+	if (unlikely(size > SLUB_MAX_SIZE))
 		return kmalloc_large_node(size, flags, node);
 
 	s = get_slab(size, flags);

@@ -3223,7 +3223,7 @@
 {
 	struct kmem_cache *s;
 
-	if (unlikely(size > PAGE_SIZE))
+	if (unlikely(size > SLUB_MAX_SIZE))
 		return kmalloc_large(size, gfpflags);
 
 	s = get_slab(size, gfpflags);

@@ -3239,7 +3239,7 @@
 {
 	struct kmem_cache *s;
 
-	if (unlikely(size > PAGE_SIZE))
+	if (unlikely(size > SLUB_MAX_SIZE))
 		return kmalloc_large_node(size, gfpflags, node);
 
 	s = get_slab(size, gfpflags);

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

:-)

Looks good to me. Yanmin, please retest netperf with this one instead if you have the time. I'll replace the revert with this patch but keep your default order tweak patch.

Pekka

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

That could be a problem, sure. Especially for architecture that have 64 K pages.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

You could likely put a complicated formula in there instead. But 2 * PAGE_SIZE is simple and will work on all platforms regardless of pagesize.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

I think 2 * PAGE_SIZE is best and the patch description is needed change. it's because almost architecture use two pages for stack and current page allocator don't have delayed consolidation mechanism for order-1 page.

In addition, if pekka patch (SLAB_LIMIT = 8K) run on ia64, 16K allocation always fallback to page allocator and using 64K (4 times memory consumption!).

Am I misunderstand anything?

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Hi!

Do you mean alloc_thread_info()? Not all architectures use kmalloc() to implement it so I'm not sure if that's relevant for this patch.

Yes, correct, but SLUB does that already by passing all allocations over 4K to the page allocator.

I'm not totally against 2 * PAGE_SIZE but I just worry that as SLUB performance will be bound to architecture page size, we will see skewed results in performance tests without realizing it. That's why I'm in favor of a fixed size that's unified across architectures.

Pekka

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Hi Pekka,

hmhm
OK. my mail was pointless.

but why? In my understanding, slab framework mainly exist for efficient sub-page allocation.
the fallbacking of 4K allocation in 64K page-sized architecture seems inefficient.

fair point.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

I don't think any of the slab allocators are known for memory efficiency. That said, the original patch description sums up the rationale for page allocator pass-through:

http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff;h=aadb4bc4a1f9108c1d0fbd121827c936c2ed4217

Interesting enough, there seems to be some performance gain from it as well as seen by Mel Gorman's recent slab allocator benchmarks.

Pekka

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

2009/2/19 Pekka Enberg <[email protected]>:

Honestly, I'm bit confusing.
above url's patch use PAGE_SIZE, but not 4K nor architecture independent value. Your 4K mean PAGE_SIZE?

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Yes, I mean PAGE_SIZE. 4K page sizes are hard-wired into my brain, sorry :-)

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Thanks!
I'm recovered from confusing.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

What could be changed in the patch is to set SLUB_MAX_SIZE depending on the page size of the underlying architecture.

#define SLUB_MAX_SIZE MAX(PAGE_SIZE, 8192)

So on 4k architectures SLUB_MAX_SIZE is set to 8192 and on 16k or 64k arches its set to PAGE_SIZE.

And then define

#define SLUB_MAX_KMALLOC_ORDER get_order(SLUB_MAX_SIZE)

which will be 1 on 4k arches and 0 on higher sized arches.

Then also the kmalloc array would need to be dimensioned using SLUB_MAX_KMALLOC_ORDER.

The definition of SLUB_NAX_KMALLOC_ORDER could be a bit challenging for the C compiler.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Kernel 2.6.29-rc2 panic with the patch.

BUG: unable to handle kernel NULL pointer dereference at (null)
IP: [<ffffffff8028fae3>] kmem_cache_alloc+0x43/0x97
PGD 0 
Oops: 0000 [#1] SMP 
last sysfs file: 
CPU 0 
Modules linked in:
Pid: 1, comm: swapper Not tainted 2.6.29-rc2slubstat8k #1
RIP: 0010:[<ffffffff8028fae3>]  [<ffffffff8028fae3>] kmem_cache_alloc+0x43/0x97
RSP: 0018:ffff88022f865e20  EFLAGS: 00010046
RAX: 0000000000000000 RBX: 0000000000000246 RCX: 0000000000000002
RDX: 0000000000000000 RSI: 000000000000063f RDI: ffffffff808096c7
RBP: 00000000000000d0 R08: 0000000000000004 R09: 000000000012e941
R10: 0000000000000002 R11: 0000000000000020 R12: ffffffff80991c48
R13: ffffffff809a9b43 R14: ffffffff809f8000 R15: 0000000000000000
FS:  0000000000000000(0000) GS:ffffffff80a13080(0000) knlGS:0000000000000000
CS:  0010 DS: 0018 ES: 0018 CR0: 000000008005003b
CR2: 0000000000000000 CR3: 0000000000201000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process swapper (pid: 1, threadinfo ffff88022f864000, task ffff88022f868000)
Stack:
 ffffffff809f43e0 0000000000000020 ffffffff809aa469 0000000000000086
 ffffffff809f8000 ffffffff809a9b43 ffffffff80aaae80 ffffffff809f43e0
 0000000000000020 ffffffff809aa469 0000000000000000 ffffffff809d86a0
Call Trace:
 [<ffffffff809aa469>] ? populate_rootfs+0x0/0xdf
 [<ffffffff809a9b43>] ? unpack_to_rootfs+0x59/0x97f
 [<ffffffff809aa469>] ? populate_rootfs+0x0/0xdf
 [<ffffffff809aa481>] ? populate_rootfs+0x18/0xdf
 [<ffffffff80209051>] ? _stext+0x51/0x120
 [<ffffffff802d69b2>] ? create_proc_entry+0x73/0x8a
 [<ffffffff802619c0>] ? register_irq_proc+0x92/0xaa
 [<ffffffff809a4896>] ? kernel_init+0x12e/0x188
 [<ffffffff8020ce3a>] ? child_rip+0xa/0x20
 [<ffffffff809a4768>] ? kernel_init+0x0/0x188
 [<ffffffff8020ce30>] ? child_rip+0x0/0x20
Code: be 3f 06 00 00 48 c7 c7 c7 96 80 80 e8 b8 e2 f9 ff e8 c5 c2 45 00 9c 5b fa 65 8b 04 25 24 00 00 00 48 98 49 8b 94 c4 e8  
RIP  [<ffffffff8028fae3>] kmem_cache_alloc+0x43/0x97
 RSP <ffff88022f865e20>
CR2: 0000000000000000
---[ end trace a7919e7f17c0a725 ]---
swapper used greatest stack depth: 5376 bytes left
Kernel panic - not syncing: Attempted to kill init!

Hi Yanmin,

Aah, we need to fix up some more PAGE_SHIFTs in the code.

Pekka

index 2f5c16b..e217a7a 100644
@@ -121,10 +121,23 @@ struct kmem_cache {
 #define KMALLOC_SHIFT_LOW ilog2(KMALLOC_MIN_SIZE)
 
 /*
+ * Maximum kmalloc object size handled by SLUB. Larger object allocations
+ * are passed through to the page allocator. The page allocator "fastpath"
+ * is relatively slow so we need this value sufficiently high so that
+ * performance critical objects are allocated through the SLUB fastpath.
+ *
+ * This should be dropped to PAGE_SIZE / 2 once the page allocator
+ * "fastpath" becomes competitive with the slab allocator fastpaths.
+ */
+#define SLUB_MAX_SIZE (2 * PAGE_SIZE)
+
+#define SLUB_PAGE_SHIFT (PAGE_SHIFT + 2)
+
+/*
  * We keep the general caches in an array of slab caches that are used for
  * 2^x bytes of allocations.
  */
-extern struct kmem_cache kmalloc_caches[PAGE_SHIFT + 1];
+extern struct kmem_cache kmalloc_caches[SLUB_PAGE_SHIFT];
 
 /*
  * Sorry that the following has to be that ugly but some versions of GCC

@@ -212,7 +225,7 @@ static __always_inline void *kmalloc_large(size_t size, gfp_t flags)
 static __always_inline void *kmalloc(size_t size, gfp_t flags)
 {
 	if (__builtin_constant_p(size)) {
-		if (size > PAGE_SIZE)
+		if (size > SLUB_MAX_SIZE)
 			return kmalloc_large(size, flags);
 
 		if (!(flags & SLUB_DMA)) {

@@ -234,7 +247,7 @@ void *kmem_cache_alloc_node(struct kmem_cache *, gfp_t flags, int node);
 static __always_inline void *kmalloc_node(size_t size, gfp_t flags, int node)
 {
 	if (__builtin_constant_p(size) &&
-		size <= PAGE_SIZE && !(flags & SLUB_DMA)) {
+		size <= SLUB_MAX_SIZE && !(flags & SLUB_DMA)) {
 			struct kmem_cache *s = kmalloc_slab(size);
 
 		if (!s)

index 0280eee..43a0c53 100644
@@ -2475,7 +2475,7 @@ EXPORT_SYMBOL(kmem_cache_destroy);
  *		Kmalloc subsystem
  *******************************************************************/
 
-struct kmem_cache kmalloc_caches[PAGE_SHIFT + 1] __cacheline_aligned;
+struct kmem_cache kmalloc_caches[SLUB_PAGE_SHIFT] __cacheline_aligned;
 EXPORT_SYMBOL(kmalloc_caches);
 
 static int __init setup_slub_min_order(char *str)

@@ -2537,7 +2537,7 @@ panic:
 }
 
 #ifdef CONFIG_ZONE_DMA
-static struct kmem_cache *kmalloc_caches_dma[PAGE_SHIFT + 1];
+static struct kmem_cache *kmalloc_caches_dma[SLUB_PAGE_SHIFT];
 
 static void sysfs_add_func(struct work_struct *w)
 {

@@ -2658,7 +2658,7 @@ void *__kmalloc(size_t size, gfp_t flags)
 {
 	struct kmem_cache *s;
 
-	if (unlikely(size > PAGE_SIZE))
+	if (unlikely(size > SLUB_MAX_SIZE))
 		return kmalloc_large(size, flags);
 
 	s = get_slab(size, flags);

@@ -2686,7 +2686,7 @@ void *__kmalloc_node(size_t size, gfp_t flags, int node)
 {
 	struct kmem_cache *s;
 
-	if (unlikely(size > PAGE_SIZE))
+	if (unlikely(size > SLUB_MAX_SIZE))
 		return kmalloc_large_node(size, flags, node);
 
 	s = get_slab(size, flags);

@@ -2986,7 +2986,7 @@ void __init kmem_cache_init(void)
 		caches++;
 	}
 
-	for (i = KMALLOC_SHIFT_LOW; i <= PAGE_SHIFT; i++) {
+	for (i = KMALLOC_SHIFT_LOW; i < SLUB_PAGE_SHIFT; i++) {
 		create_kmalloc_cache(&kmalloc_caches[i],
 			"kmalloc", 1 << i, GFP_KERNEL);
 		caches++;

@@ -3023,7 +3023,7 @@ void __init kmem_cache_init(void)
 	slab_state = UP;
 
 	/* Provide the correct kmalloc names now that the caches are up */
-	for (i = KMALLOC_SHIFT_LOW; i <= PAGE_SHIFT; i++)
+	for (i = KMALLOC_SHIFT_LOW; i < SLUB_PAGE_SHIFT; i++)
 		kmalloc_caches[i]. name =
 			kasprintf(GFP_KERNEL, "kmalloc-%d", 1 << i);
 

@@ -3223,7 +3223,7 @@ void *__kmalloc_track_caller(size_t size, gfp_t gfpflags, unsigned long caller)
 {
 	struct kmem_cache *s;
 
-	if (unlikely(size > PAGE_SIZE))
+	if (unlikely(size > SLUB_MAX_SIZE))
 		return kmalloc_large(size, gfpflags);
 
 	s = get_slab(size, gfpflags);

@@ -3239,7 +3239,7 @@ void *__kmalloc_node_track_caller(size_t size, gfp_t gfpflags,
 {
 	struct kmem_cache *s;
 
-	if (unlikely(size > PAGE_SIZE))
+	if (unlikely(size > SLUB_MAX_SIZE))
 		return kmalloc_large_node(size, gfpflags, node);
 
 	s = get_slab(size, gfpflags);

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

The new patch fixes hang issue. netperf UDP-U-4k (start CPU_NUM clients) result is pretty good.

Do you have your patch on top of it as well? Btw, can I add a Tested-by tag from you to the patch?

Pekka

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Yes.

Ok. Another testing with UDP-U-4k (start 1 client and bind client and server to different cpu) result is improved, but is not so good as SLQB's. But we can increase slub_max_order to get the similiar result like SLQB.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

This is merged now with my fixlets:

http://git.kernel.org/?p=linux/kernel/git/penberg/slab-2.6.git;a=commitdiff;h=8573e12414365585bfd601dc8c093b3efbef8854

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Hi Mel,

Did you just cherry-pick the patch or did you run it with the topic/slub/perf branch? There's a follow-up patch from Yanmin which will make a difference for large allocations when page-allocator pass-through is reverted:

http://git.kernel.org/?p=linux/kernel/git/penberg/slab-2.6.git;a=commitdiff;h=79b350ab63458ef1d11747b4f119baea96771a6e

Pekka

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Cherry picked to minimise the number of factors involved.

Is this expected to make a difference to workloads that are not that allocator intensive? I doubt it'll make much different to speccpu but conceivably it makes a difference to sysbench.

Mel Gorman
Part-time Phd Student                          Linux Technology Center
University of Limerick                         IBM Dublin Software Lab

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

I doubt that too but I fail to see why it's regressing with the revert in the first place for speccpu. Maybe it's cache effects, dunno.

Pekka

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Just to clarify on this last point, do you mean slub_max_order=0 to force order-0 allocations in SLUB?

Mel Gorman
Part-time Phd Student                          Linux Technology Center
University of Limerick                         IBM Dublin Software Lab

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Hmm... I think slub_min_objects=1 should also do basically the same. Actually slub_min_object=1 and slub_max_order=1 should get closest I think.

I'm going with slub_min_objects=1 and slub_max_order=0. A quick glance of the source shows the calculation as

        for (order = max(min_order,
                                fls(min_objects * size - 1) - PAGE_SHIFT);
                        order <= max_order; order++) {

so the max_order is inclusive not exclusive. This will force the order-0 allocations I think you are looking for.

Mel Gorman
Part-time Phd Student                          Linux Technology Center
University of Limerick                         IBM Dublin Software Lab

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Well, but in the case of really bad internal fragmentation in the page, SLAB will do order-1 allocations even if it doesn't strictly need to. Probably this isn't a huge deal, but I think if we do slub_min_objects=1, then SLUB won't care about number of objects per page, and slub_max_order=1 will mean it stops caring about fragmentation after order-1. I think. Which would be pretty close to SLAB (depending on exactly how much fragmentation it cares about).

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Ok, very good point and I agree with your assessment. Tests are restarted with slub_min_objects=1 slab_max_order=1. The dmesg line related to SLUB looks like

SLUB: Genslabs=13, HWalign=64, Order=0-1, MinObjects=1, CPUs=4, Nodes=1

Mel Gorman
Part-time Phd Student                          Linux Technology Center
University of Limerick                         IBM Dublin Software Lab

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

slub_max_order=0 will fore all possible slabs to order 0. This means that some slabs that SLAB will run as order 1 will be order 0 under SLUB.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

slub_max_order=0 would be sufficient.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Hi Mel,

Yup. That's more or less what I've been saying for a long time now. The OLTP regression is not all obvious and while there has been plenty of talk about it (cache line ping-pong due to lack of queues, high order pages), I've yet to see a detailed analysis on it.

It would be interesting to know what drivers the Intel setup uses. One thing I speculated with Christoph at OLS is that the regression could be due to bad interaction with the SCSI subsystem, for example. That would explain why the regression doesn't show up in typical setups which have ATA.

Anyway, even if we did end up going forward with SLQB, it would sure as hell be less painful if we understood the reasons behind it.

Pekka

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

The reasons may depend on hardware contingencies like TLB handling overhead and various inefficiencies that depend on the exact processor model. Also the type of applications you want to run. Some of the IA64 heritage of SLUB may be seen in the results of these tests. Note that PPC and IA64 have larger page sizes (which results in SLUB being able to put more objects into an order 0 page) and higher penalties for TLB handling. The initial justification for SLUB were Mel's results on IA64 that showed a 5-10% increase in performance through SLUB.

In my current position we need to run extremely low latency code in user space and want to avoid any disturbance by kernel code interrupting user space. My main concern for my current work context is that switching to SLQB will bring back the old cache cleaning problems and introduce latencies for our user space applications. Otherwise I am on x86 now so the TLB issues are less of a concern for me now.

In general it may be better to have a larger selection of slab allocators. I think this is no problem as long as we have motivated people that maintain these. Nick seems to be very motivated at this point. So lets merge SLQB as soon as we can and expose it to a wider audience so that it can mature. And people can have more fun running one against the other refining these more and more.

There are still two major things that I hope will happen soon to clean up stuff in the slab allocators:

1. The introduction of a per cpu allocator.

This is important to optimize the fastpaths. The cpu allocator will allow us to get rid of the arrays indexes by NR_CPUS and allow operations that are atomic wrt. interrupts. The lookup of the kmem_cache_cpu struct address will no longer be necessary.

2. Alloc/free without disabling interrupts.

Matthieu has written an early implementation of allocation functions that do not require interrupt disable/enable. It seems that these are right now the major cause of latency in the fast paths. Andi has stated that the interrupt enable/disable has been optimized in recent releases of new processors. The overhead may be due to the flags being pushed onto the stack and retrieved later. Mathieus implementation can be made more elegant if atomic per cpu ops are available. This could significantly increase the speed of the fast paths in the allocators (may be a challenge to SLAB and SLQB since they need to update a counter and a pointer but its straightforward in SLUB).

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>