Skip to contentSkip to conversation

Modular slab allocator V3

PATCH In review Christoph Lameter 1 participant 7 messages Sep 1, 2006

Subsystems (4)
lore
Share & tools
lore
#0 lore raw

cover Modular slab allocator V3

Modular Slab Allocator:

Why would one use this?

1. Reduced memory requirements.

Saving range from a few hundred kbyte on i386 to 5GB on a 1024p 4TB Altix NUMA system.

The slabifier has no caches in the sense of the slab allocator. No storage is allocated for per cpu, shared or alien caches. A slab in itself functions as the cache. Objects are served directly from a per cpu slab (an "active" slab). The management overhead for caches is gone.

Slabs do not contain metadata but only the payload. Metadata is kept in the associated page struct. This means that object can begin at the start of a slab and are always properly aligned.

2. No cache reaper

The current slab allocator needs to periodically check its slab caches and move objects back into the slabs. Every 2 seconds on every cpu all slab caches are scanned and object move around the system. The system cannot really enter a quiescent state.

The slabifier needs no such mechanism in the single processor case. In the SMP case we have a per slab flusher that is active as long as processors have active slabs. After a timeout it flushes the active slabs back into the slab lists. If no active slabs exist then the flusher is deactivated.

The cache_reaper has been a consistent trouble spot for interrupt holdoffs and scheduling latencies in the SLES9 and SLES10 development cycle. I would be grateful if we would not have to deal again with that.

3. Can use the NUMA policies of the page allocator.

The current slab allocator implements NUMA support through per node lists of slabs. If a memory policy or cpuset restrict access to certain node then the slab allocator itself must enforce these policies. This is only partially implemented and as a result cpusets, memory policies and the NUMA slab do not mix too well which leads to per node slabs containing slabs that are actually located on different nodes, which causes latencies during cache draining (in the cache reaper and elsewhere....).

The Slabifier does not implement per node slabs. Instead it uses a single global pool of partial pages. Memory policies only come into play when the active slab is empty and new pages are allocated from the page allocator. In that case the page is allocated given the current cpuset and the current memory policies by the page allocator and then the slabifier serves objects from the slab to the application. The Slabifier does not attempt to guarantee that the allocations by kmalloc() are node local. It only does a best effort approach. Only kmalloc_node() guarantees that an object is allocated on a particular node. kmalloc_node() accomplishes that by searching the partial list for a fitting page and allocating a page from the requested node if none can be found.

4. Reduced amount of partial slabs.

The current NUMA slab allocator contains per node partial lists. This means that fragmentation of slabs occurs on a per node basis. The more nodes are in the system the more potentially partially allocated slabs.

The slabifier contains a global partial lists. Allocations on other nodes can cause the partial list to be shrunk. The existing slab pages of a slab cache have therefore a higher usage rate.

The locking of the partial list is potential scalability problem that is addressed in the following ways:

A. The partial list is only modified (and the lock taken) when necessary.
Locking is only necessary when a page enters the partial list (it was full and the first object was deleted), or it becomes completely depleted (slab has to be freed) or it is retrieved to become an active slab for allocations of a particular cpu.

B. A "min_slab_order=" kernel boot option is added. This allows to increase
the size of the slab pages. Bigger slabs mean less lock taking and larger per cpu caches. It also reduces slab fragmentation but comes with the danger that the kernel cannot satisfy higher order allocations. However, order 1,2,3 allocations should usually be fine. In my tests up to 32p I have not yet seen a need to use this to reduce lock contention.

5. Ability to compactify partial lists.

The slab shrinker of the slabifier can take an argument of a function that is capable of moving an object. With that the slabifier is able to reduce the amount of partial slabs.

6. Maintainability,

The Modular Slab is made up out of components that can individually be replaced. Modifications are easy and it is easier to add new features.

7. Performance

The performance of the Modular Slab is roughly comparable with the existing slab allocator since both rely on managing a per cpu cache of objects. The slab allocator does that by explicitly managing object lists and the slabifier does it by reserving a slab per cpu for allocations.

TODO:

- More performance tests than just with AIM7.... Higher CPU counts than 32.

Changes V2->V3:

- Tested on i386 (UP + SMP) , x86_64(up), IA64(NUMA up to 32p)

- Overload struct page in mm.h with slab definitions. That reduces the macros significantly and makes code more readable.

- Debug and optimize functions, Reduce cacheline footprint.

- Add support for specifying slab_min_order= at bootup in order to be able to influence fragmentation and lock scaling.

Changes V1-V2:
- Drop pageslab and numaslab. Drop support for VMALLOC allocations.

- Enhance slabifier with some numa capability. Bypass free list management for slabs with a single object. Drop slab full lists and minimize lock taking for partial lists.

- Optimize code: Generate general slab array immediately and pass the address of the slab cache in kmalloc(). DMA caches remain dynamic.

- Add support for non power of 2 general caches.

- Tested on i386, x86_64 and ia64.

The main intend of this patchset is to modularize the slab allocator so that development of additions or modification to the allocator layer become easier. The framework enables the use of multiple slab allocator and allows the generation of additional underlying page allocators (as f.e. needed for mempools and other specialized things).

The modularization is accomplished through the use of a few concepts from object oriented programming. Allocators are described by methods and functions can produce new allocators based on existing ones by modifying their methods.

So what the patches provide here is:

1. A framework for page allocators and slab allocators

2. Various methods to derive new allocators from old ones (add rcu support, destructors, constructors, dma etc)

3. A layer that emulates the exist slab interface (the slabulator).

4. A layer that provides kmalloc functionality.

5. The Slabifier. This is conceptually the Simple Slab (See my RFC from last week) but with the additional allocator modifications possible it grows like on steroids and then can supply most of the functionality of the existing slab allocator and can go even beyond it. My tests with AIM7 seem indicates that it is equal in performance to the existing slab allocator.

Some new features:

1. The slabifier can flag double frees when the act occurs and will attempt to continue.

2. Ability to merge slabs of the same type.

Notably missing features:

- Slab Debugging
(This should be implemented by deriving a new slab allocator from existing ones and adding the necessary processing in alloc and free).

Performance tests on an 8p and 32p machine show consistently that the performance is equal to the standard slab allocator. Memory use is much less with this since there is no meta data overhead per slab.

This patchset should just leave the existing slab allocator unharmed. It adds a hook to include/linux/slab.h to redirect includes to the definitions for the allocation framework by the slabulator.

Deactivate the "Traditional Slab allocator" in order to activate the modular slab allocator.

More details may be found in the header of each of the following 4 patches.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#1 lore raw

PATCH 1/5 Generic Allocator Framework

Add allocator abstraction

The allocator abstraction layer provides sources of pages for the slabifier and it provides ways to customize the slabifier to ones needs (one can put dmaificiation, rcuification and so on of slab frees etc on top of the standard page allocator).

The allocator framework also provides a means for deriving new slab allocators from old ones. That way features can be added in a generic way. It would be possible to add rcu for slab objects or debugging in that fashion.

The object-oriented style of deconstructing the allocators has the advantage that we can deal with small pieces of code that add special functionality. The overall framework makes it easy to replace pieces and evolve the whole allocator systems in a faster way.

It also provides a generic way to operate on different allocators.

It is no problem to define a new allocator that allocates from memory pools and then use the slab allocator on that memory pool.

The code in mm/allocators.c provides some examples what could be done with derived allocators.

Signed-off-by Christoph Lameter <clameter>.
mm/Makefile +1 -0
Index: linux-2.6.18-rc5-mm1/mm/Makefile
===================================================================
@@ -28,3 +28,4 @@ obj-$(CONFIG_MEMORY_HOTPLUG) += memory_h
 obj-$(CONFIG_FS_XIP) += filemap_xip.o
 obj-$(CONFIG_MIGRATION) += migrate.o
 obj-$(CONFIG_SMP) += allocpercpu.o
+obj-$(CONFIG_MODULAR_SLAB) += allocator.o
include/linux/allocator.h new file +221 -0

221 lines not on this page · raw message

mm/allocator.c new file +451 -0

451 lines not on this page · raw message

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#2 lore raw

PATCH 2/5 Slabifier

In reply to Christoph Lameter

V2->V3:
- Overload struct page
- Add new PageSlabsingle flag

Lately I have started tinkering around with the slab in particular after Matt Mackal mentioned that the slab should be more modular at the KS. One particular design issue with the current slab is that it is build on the basic notion of shifting object references from list to list. Without NUMA this is wild enough with the per cpu caches and the shared cache but with NUMA we now have per node shared arrays, per node list and per node per node alien caches. Somehow this all works but one wonders does it have to be that way? On very large systems the number of these entities grows to unbelievable numbers. On our 1k cpu/node system each slab need 128M for alien caches alone.

So I thought it may be best to try to develop another basic slab layer that does not have all the object queues and that does not have to carry so much state information. I also have had concerns about the way locking is handled for awhile. We could increase parallelism by finer grained locking. This in turn may avoid the need for object queues.

One of the problems of the NUMA slab allocator is that per node partial slab lists are used. Partial slabs cannot be filled up from other nodes. So what I have tried to do here is to have minimal metainformation combined with one centralized list of partially allocated slabs. The list_lock is only taken if list modifications become necessary. The need for those has been drastically reduced with a few measures. See below.

After toying around for awhile I came to the realization that the page struct contains all the information necessary to manage a slab block. One can put all the management information there and that is also advantageous for performance since we constantly have to use the page struct anyways for reverse object lookups and during slab creation. So this also reduces the cache footprint of the slab. The alignment is naturally the best since the first object starts right at the page boundary. This reduces the complexity of alignment calculations.

Also we have a page lock in the page struct that is used for locking each slab during modifications. Taking the lock per slab is the finest grained locking available and this is fundamental to the slabifier. The slab lock is taken if the slab contains multiple objects in order to protect the freelist.

The freelists of objects per page are managed as a chained list. The struct page contains a pointer to the first element. The first 4 bytes of the free element contains a pointer to the next free element etc until the chain ends with NULL. If the object cannot be overwritten after free (RCU and constructors etc) then we can shift the pointer to the next free element behind the object.

The slabifier does remove the need for a list of free slabs and a list of used slabs. Free slabs are immediately returned to the page allocator. The page allocator has its own per cpu queues that will manage these free pages. Used slabs are simply not tracked because we never have a need to find out where the used slabs are. The only important thing is that the slabs come back to the partial list when an object in them is deleted. The metadata in the corresponding page struct will allow us to do that easily.

Per cpu caches exist in the sense that each processor has a per processor "cpuslab". Objects in this "active" slab will only be allocated from this processor. This naturally makes all allocations from a single processor come from the same slab page which reduces fragmentation.
The page state is likely going to stay in the cache. Allocation will be very fast since we only need the page struct reference for all our needs which is likely not contended at all. Fetching the next free pointer from the location of the object nicely prefetches the object.

The list_lock is used only in very rare cases. Lets discuss one example of multiple processors allocating from the same cache. The first thing that happens when the slab cache is empty is that every processors gets its own slab (the "active" slab). This does not require the list_lock because we get a page from the page allocator and that immediately becomes the active slab. Now all processors allocate from their slabs. This also does not require any access to the partial lists so no list_lock is taken.

If a slab becomes full then each processor is simply forgetting about the slab and gets a new one from the page allocator.

Therefore as long as all processors are just allocating no list_lock is needed at all.

If a free now happens then things get a bit more complicated. If the free occurs on an active page then again no list_lock needs to be taken. Only the slab lock may be contended since it may be under current allocation by a process.

If the free occurs on a fully allocated page then we make a partially allocated page from a full page. Now the list_lock will be taken and the page is put on the partial list.

If further frees occur on a partially allocated page then also no list_lock needs to be taken because it is still a partially allocated page. This works until the page has no objects left. At that point we take the page of the list of partial slabs to free it and that requires the list_lock again.

If a processors has filled up its active slab and needs a new one then it will first check if there are partially allocated slabs available. If so then it will take a partially allocated slab and begin to fill it up. That also requires taking the list lock.

mm/slabifier.c new file +936 -0
Index: linux-2.6.18-rc5-mm1/mm/slabifier.c
===================================================================
@@ -0,0 +1,936 @@
+/*
+ * Generic Slabifier for the allocator abstraction framework.
+ *
+ * The allocator synchronizes using slab based locks and only
+ * uses a centralized list lock to manage the pool of partial slabs.
+ *
+ * (C) 2006 Silicon Graphics Inc., Christoph Lameter <[email protected]>
+ */
+
+#include <linux/mm.h>
+#include <linux/allocator.h>
+#include <linux/bit_spinlock.h>
+#include <linux/module.h>
+#include <linux/interrupt.h>
+
+/*
+ * Enabling SLABIFIER_DEBUG will check various things. Among others it will
+ * flag double frees.
+ */
+/* #define SLABIFIER_DEBUG */
+
+struct slab {
+	struct slab_cache sc;
+#ifdef CONFIG_SMP
+	int flusher_active;
+	struct work_struct flush;
+#endif
+	atomic_t refcount;		/* Refcount for destroy */
+	atomic_long_t nr_slabs;		/* Total slabs used */
+	/* Performance critical items follow */
+	int size;			/* Total size of an object */
+	int offset;			/* Free pointer offset. */
+	int objects;			/* Number of objects in slab */
+	spinlock_t list_lock;
+	struct list_head partial;
+	unsigned long nr_partial;
+	struct page *active[NR_CPUS];
+};
+
+/*
+ * The page struct is used to keep necessary information about a slab.
+ * For a compound page the first page keeps the slab state.
+ *
+ * Lock order:
+ *   1. slab_lock(page)
+ *   2. slab->list_lock
+ *
+ * The slabifier assigns one slab for allocation to each processor.
+ * Allocations only occur from these active slabs.
+ *
+ * If a cpu slab is active then a workqueue thread checks every 10
+ * seconds if the cpu slab is still in use. The cpu slab is pushed back
+ * to the list if inactive [only needed for SMP].
+ *
+ * Leftover slabs with free elements are kept on a partial list.
+ * There is no list for full slabs. If an object in a full slab is
+ * freed then the slab will show up again on the partial lists.
+ * Otherwise there is no need to track filled up slabs.
+ *
+ * Slabs are freed when they become empty. Teardown and setup is
+ * minimal so we rely on the page allocators per cpu caches for
+ * fast frees and allocations.
+ */
+
+/*
+ * Locking for each individual slab using the pagelock
+ */
+static __always_inline void slab_lock(struct page *page)
+{
+	bit_spin_lock(PG_locked, &page->flags);
+}
+
+static __always_inline void slab_unlock(struct page *page)
+{
+	bit_spin_unlock(PG_locked, &page->flags);
+}
+
+/*
+ * Management of partially allocated slabs
+ */
+static void __always_inline add_partial(struct slab *s, struct page *page)
+{
+	spin_lock(&s->list_lock);
+	s->nr_partial++;
+	list_add_tail(&page->lru, &s->partial);
+	spin_unlock(&s->list_lock);
+}
+
+static void __always_inline remove_partial(struct slab *s,
+						struct page *page)
+{
+	spin_lock(&s->list_lock);
+	list_del(&page->lru);
+	s->nr_partial--;
+	spin_unlock(&s->list_lock);
+}
+
+/*
+ * Lock page and remove it from the partial list
+ *
+ * Must hold list_lock
+ */
+static __always_inline int lock_and_del_slab(struct slab *s,
+						struct page *page)
+{
+	if (bit_spin_trylock(PG_locked, &page->flags)) {
+		list_del(&page->lru);
+		s->nr_partial--;
+		return 1;
+	}
+	return 0;
+}
+
+/*
+ * Get a partial page, lock it and return it.
+ */
+#ifdef CONFIG_NUMA
+static struct page *get_partial(struct slab *s, int node)
+{
+	struct page *page;
+	int searchnode = (node == -1) ? numa_node_id() : node;
+
+	spin_lock(&s->list_lock);
+	/*
+	 * Search for slab on the right node
+	 */
+	list_for_each_entry(page, &s->partial, lru)
+		if (likely(page_to_nid(page) == searchnode) &&
+			lock_and_del_slab(s, page))
+				goto out;
+
+	if (likely(node == -1)) {
+		/*
+		 * We can fall back to any other node in order to
+		 * reduce the size of the partial list.
+		 */
+		list_for_each_entry(page, &s->partial, lru)
+			if (likely(lock_and_del_slab(s, page)))
+				goto out;
+	}
+
+	/* Nothing found */
+	page = NULL;
+out:
+	spin_unlock(&s->list_lock);
+	return page;
+}
+#else
+static struct page *get_partial(struct slab *s, int node)
+{
+	struct page *page;
+
+	spin_lock(&s->list_lock);
+	list_for_each_entry(page, &s->partial, lru)
+		if (likely(lock_and_del_slab(s, page)))
+			goto out;
+
+	/* No slab or all slabs busy */
+	page = NULL;
+out:
+	spin_unlock(&s->list_lock);
+	return page;
+}
+#endif
+
+
+/*
+ * Debugging checks
+ */
+static void check_slab(struct page *page)
+{
+#ifdef SLABIFIER_DEBUG
+	if (!PageSlab(page)) {
+		printk(KERN_CRIT "Not a valid slab page @%p flags=%lx"
+			" mapping=%p count=%d \n",
+			page, page->flags, page->mapping, page_count(page));
+		BUG();
+	}
+#endif
+}
+
+static int check_valid_pointer(struct slab *s, struct page *page,
+					 void *object, void *origin)
+{
+#ifdef SLABIFIER_DEBUG
+	void *base = page_address(page);
+
+	if (object < base || object >= base + s->objects * s->size) {
+		printk(KERN_CRIT "slab %s size %d: pointer %p->%p\nnot in"
+			" range (%p-%p) in page %p\n", s->sc.name, s->size,
+			origin, object, base, base + s->objects * s->size,
+			page);
+		return 0;
+	}
+
+	if ((object - base) % s->size) {
+		printk(KERN_CRIT "slab %s size %d: pointer %p->%p\n"
+			"does not properly point"
+			"to an object in page %p\n",
+			s->sc.name, s->size, origin, object, page);
+		return 0;
+	}
+#endif
+	return 1;
+}
+
+/*
+ * Determine if a certain object on a page is on the freelist and
+ * therefore free. Must hold the slab lock for active slabs to
+ * guarantee that the chains are consistent.
+ */
+static int on_freelist(struct slab *s, struct page *page, void *search)
+{
+	int nr = 0;
+	void **object = page->freelist;
+	void *origin = &page->lru;
+
+	if (PageSlabsingle(page))
+		return 0;
+
+	check_slab(page);
+
+	while (object && nr <= s->objects) {
+		if (object == search)
+			return 1;
+		if (!check_valid_pointer(s, page, object, origin))
+			goto try_recover;
+		origin = object;
+		object = object[s->offset];
+		nr++;
+	}
+
+	if (page->inuse != s->objects - nr) {
+		printk(KERN_CRIT "slab %s: page %p wrong object count."
+			" counter is %d but counted were %d\n",
+			s->sc.name, page, page->inuse,
+			s->objects - nr);
+try_recover:
+		printk(KERN_CRIT "****** Trying to continue by marking "
+			"all objects in the slab used (memory leak!)\n");
+		page->inuse = s->objects;
+		page->freelist =  NULL;
+	}
+	return 0;
+}
+
+void check_free_chain(struct slab *s, struct page *page)
+{
+#ifdef SLABIFIER_DEBUG
+	on_freelist(s, page, NULL);
+#endif
+}
+
+/*
+ * Operations on slabs
+ */
+static void discard_slab(struct slab *s, struct page *page)
+{
+	atomic_long_dec(&s->nr_slabs);
+
+	page->mapping = NULL;
+	reset_page_mapcount(page);
+	__ClearPageSlab(page);
+	__ClearPageSlabsingle(page);
+
+	s->sc.page_alloc->free(s->sc.page_alloc, page, s->sc.order);
+}
+
+/*
+ * Allocate a new slab and prepare an empty freelist and the basic struct
+ * page settings.
+ */
+static struct page *new_slab(struct slab *s, gfp_t flags, int node)
+{
+	struct page *page;
+
+	if (flags & __GFP_WAIT)
+		local_irq_enable();
+
+	page = s->sc.page_alloc->allocate(s->sc.page_alloc, s->sc.order,
+			flags, node == -1 ? s->sc.node : node);
+	if (!page)
+		goto out;
+
+	page->offset = s->offset;
+
+	atomic_long_inc(&s->nr_slabs);
+
+	page->slab = (struct slab_cache *)s;
+	__SetPageSlab(page);
+
+	if (s->objects > 1) {
+		void *start = page_address(page);
+		void *end = start + s->objects * s->size;
+		void **last = start;
+		void *p = start + s->size;
+
+		while (p < end) {
+			last[s->offset] = p;
+			last = p;
+			p += s->size;
+		}
+		last[s->offset] = NULL;
+		page->freelist = start;
+		page->inuse = 0;
+		check_free_chain(s, page);
+	} else
+		__SetPageSlabsingle(page);
+
+out:
+	if (flags & __GFP_WAIT)
+		local_irq_disable();
+	return page;
+}
+
+/*
+ * Move a page back to the lists.
+ *
+ * Must be called with the slab lock held.
+ *
+ * On exit the slab lock will have been dropped.
+ */
+static void __always_inline putback_slab(struct slab *s, struct page *page)
+{
+	if (page->inuse) {
+		if (page->inuse < s->objects)
+			add_partial(s, page);
+		slab_unlock(page);
+	} else {
+		slab_unlock(page);
+		discard_slab(s, page);
+	}
+}
+
+/*
+ * Remove the currently active slab
+ */
+static void __always_inline deactivate_slab(struct slab *s,
+						struct page *page, int cpu)
+{
+	s->active[cpu] = NULL;
+	__ClearPageActive(page);
+	__ClearPageReferenced(page);
+
+	putback_slab(s, page);
+}
+
+/*
+ * Deactivate slab if we have an active slab.
+ */
+static void flush_active(struct slab *s, int cpu)
+{
+	struct page *page;
+	unsigned long flags;
+
+	local_irq_save(flags);
+	page = s->active[cpu];
+	if (likely(page)) {
+		slab_lock(page);
+		deactivate_slab(s, page, cpu);
+	}
+	local_irq_restore(flags);
+}
+
+#ifdef CONFIG_SMP
+/*
+ * Check active per cpu slabs and flush them if they are not in use.
+ */
+void flusher(void *d)
+{
+	struct slab *s = d;
+	int cpu = smp_processor_id();
+	struct page *page;
+	int nr_active = 0;
+
+	for_each_online_cpu(cpu) {
+
+		page = s->active[cpu];
+		if (!page)
+			continue;
+
+		if (PageReferenced(page)) {
+			ClearPageReferenced(page);
+			nr_active++;
+		} else
+			flush_active(s, cpu);
+	}
+	if (nr_active)
+		schedule_delayed_work(&s->flush, 2 * HZ);
+	else
+		s->flusher_active = 0;
+}
+
+static void drain_all(struct slab *s)
+{
+	int cpu;
+
+	if (s->flusher_active) {
+		cancel_delayed_work(&s->flush);
+		for_each_possible_cpu(cpu)
+			flush_active(s, cpu);
+		s->flusher_active = 0;
+	}
+}
+#else
+static void drain_all(struct slab *s)
+{
+	flush_active(s, 0);
+}
+#endif
+
+static __always_inline void *__slab_alloc(struct slab_cache *sc,
+					gfp_t gfpflags, int node)
+{
+	struct slab *s = (void *)sc;
+	struct page *page;
+	void **object;
+	void *next_object;
+	unsigned long flags;
+	int cpu;
+
+	local_irq_save(flags);
+	cpu = smp_processor_id();
+	page = s->active[cpu];
+	if (!page)
+		goto new_slab;
+
+	slab_lock(page);
+	check_free_chain(s, page);
+	if (unlikely(!page->freelist))
+		goto another_slab;
+
+	if (unlikely(node != -1 && page_to_nid(page) != node))
+		goto another_slab;
+redo:
+	page->inuse++;
+	object = page->freelist;
+	page->freelist = next_object = object[page->offset];
+	__SetPageReferenced(page);
+	slab_unlock(page);
+	local_irq_restore(flags);
+	return object;
+
+another_slab:
+	deactivate_slab(s, page, cpu);
+
+new_slab:
+	/*
+	 * This was moved out of line since it dereferences s and thus
+	 * potentially touches an extra cacheline
+	 */
+	if (unlikely(s->objects == 1)) {
+		page = new_slab(s, gfpflags, node);
+		local_irq_restore(flags);
+		if (page)
+			return page_address(page);
+		else
+			return NULL;
+	}
+
+	/* Racy check. If we mistakenly see no partial slabs then we
+	 * just allocate an empty slab. If we mistakenly try to get a
+	 * partial slab then get_partials() will return NULL.
+	 */
+	if (s->nr_partial) {
+		page = get_partial(s, node);
+		if (page)
+			goto gotpage;
+	}
+
+	page = new_slab(s, flags, node);
+
+	if (!page) {
+		local_irq_restore(flags);
+		return NULL;
+	}
+
+	slab_lock(page);
+
+gotpage:
+	if (s->active[cpu]) {
+		slab_unlock(page);
+		discard_slab(s, page);
+		page = s->active[cpu];
+		slab_lock(page);
+	} else
+		s->active[cpu] = page;
+
+	__SetPageActive(page);
+	check_free_chain(s, page);
+
+#ifdef CONFIG_SMP
+	if (keventd_up() && !s->flusher_active) {
+		s->flusher_active = 1;
+		schedule_delayed_work(&s->flush, 2 * HZ);
+	}
+#endif
+	goto redo;
+}
+
+static void *slab_alloc(struct slab_cache *sc, gfp_t gfpflags)
+{
+	return __slab_alloc(sc, gfpflags, -1);
+}
+
+static void *slab_alloc_node(struct slab_cache *sc, gfp_t gfpflags,
+							int node)
+{
+#ifdef CONFIG_NUMA
+	return __slab_alloc(sc, gfpflags, node);
+#else
+	return slab_alloc(sc, gfpflags);
+#endif
+}
+
+static void slab_free(struct slab_cache *sc, const void *x)
+{
+	struct slab *s = (void *)sc;
+	struct page * page;
+	void *prior;
+	void **object = (void *)x;
+	unsigned long flags;
+
+	if (!object)
+		return;
+
+	page = virt_to_page(x);
+
+	if (unlikely(PageCompound(page)))
+		page = page->first_page;
+
+	if (!s)
+		s = (void *)page->slab;
+
+	if (unlikely(PageSlabsingle(page)))
+		goto single_object_slab;
+
+#ifdef SLABIFIER_DEBUG
+	if (unlikely(s != (void *)page->slab))
+		goto slab_mismatch;
+	if (unlikely(!check_valid_pointer(s, page, object, NULL)))
+		goto dumpret;
+#endif
+
+	local_irq_save(flags);
+	slab_lock(page);
+
+#ifdef SLABIFIER_DEBUG
+	if (on_freelist(s, page, object))
+		goto double_free;
+#endif
+
+	prior = object[page->offset] = page->freelist;
+	page->freelist = object;
+	page->inuse--;
+
+	if (likely(PageActive(page) || (page->inuse && prior))) {
+out_unlock:
+		slab_unlock(page);
+		local_irq_restore(flags);
+		return;
+	}
+
+	if (!prior) {
+		/*
+		 * Page was fully used before. It will have one free
+		 * object now. So move to the partial list.
+		 */
+		add_partial(s, page);
+		goto out_unlock;
+	}
+
+	/*
+	 * All object have been freed.
+	 */
+	remove_partial(s, page);
+	slab_unlock(page);
+	discard_slab(s, page);
+	local_irq_restore(flags);
+	return;
+
+single_object_slab:
+	discard_slab(s, page);
+	return;
+
+#ifdef SLABIFIER_DEBUG
+double_free:
+	printk(KERN_CRIT "slab_free %s: object %p already free.\n",
+					s->sc.name, object);
+	dump_stack();
+	goto out_unlock;
+
+slab_mismatch:
+	if (!PageSlab(page)) {
+		printk(KERN_CRIT "slab_free %s size %d: attempt to free "
+			"object(%p) outside of slab.\n",
+			s->sc.name, s->size, object);
+		goto dumpret;
+	}
+
+	if (!page->slab) {
+		printk(KERN_CRIT
+			"slab_free : no slab(NULL) for object %p.\n",
+					object);
+			goto dumpret;
+	}
+
+	printk(KERN_CRIT "slab_free %s(%d): object at %p"
+			" belongs to slab %s(%d)\n",
+			s->sc.name, s->sc.size, object,
+			page->slab->name, page->slab->size);
+
+dumpret:
+	dump_stack();
+	printk(KERN_CRIT "***** Trying to continue by not "
+			"freeing object.\n");
+	return;
+#endif
+}
+
+/* Figure out on which slab object the object resides */
+static __always_inline struct page *get_object_page(const void *x)
+{
+	struct page * page = virt_to_page(x);
+
+	if (unlikely(PageCompound(page)))
+		page = page->first_page;
+
+	if (!PageSlab(page))
+		return NULL;
+
+	return page;
+}
+
+/*
+ * slab_create produces objects aligned at size and the first object
+ * is placed at offset 0 in the slab (We have no metainformation on the
+ * slab, all slabs are in essence off slab).
+ *
+ * In order to get the desired alignment one just needs to align the
+ * size.
+ *
+ * Notice that the allocation order determines the sizes of the per cpu
+ * caches. Each processor has always one slab available for allocations.
+ * Increasing the allocation order reduces the number of times that slabs
+ * must be moved on and off the partial lists and therefore may influence
+ * locking overhead.
+ *
+ * The offset is used to relocate the free list link in each object. It is
+ * therefore possible to move the free list link behind the object. This
+ * is necessary for RCU to work properly and also useful for debugging.
+ *
+ * However no freelists are necessary if there is only one element per
+ * slab.
+ */
+static struct slab_cache *slab_create(struct slab_control *x,
+	const struct slab_cache *sc)
+{
+	struct slab *s = (void *)x;
+	int cpu;
+
+	/* Verify that the generic structure is big enough for our data */
+	BUG_ON(sizeof(struct slab_control) < sizeof(struct slab));
+
+	memcpy(&x->sc, sc, sizeof(struct slab_cache));
+
+	s->size = ALIGN(sc->size, sizeof(void *));
+
+	if (sc->offset > s->size - sizeof(void *) ||
+			(sc->offset % sizeof(void*)))
+		return NULL;
+
+	s->offset = sc->offset / sizeof(void *);
+	BUG_ON(s->offset > 65535);
+	s->objects = (PAGE_SIZE << sc->order) / s->size;
+	BUG_ON(s->objects > 65535);
+	atomic_long_set(&s->nr_slabs, 0);
+	s->nr_partial = 0;
+#ifdef CONFIG_SMP
+	s->flusher_active = 0;
+	INIT_WORK(&s->flush, &flusher, s);
+#endif
+	if (!s->objects)
+		return NULL;
+
+	INIT_LIST_HEAD(&s->partial);
+
+	atomic_set(&s->refcount, 1);
+	spin_lock_init(&s->list_lock);
+
+	for_each_possible_cpu(cpu)
+		s->active[cpu] = NULL;
+	return &s->sc;
+}
+
+/*
+ * Check if a given pointer is valid
+ */
+static int slab_pointer_valid(struct slab_cache *sc, const void *object)
+{
+	struct slab *s = (void *)sc;
+	struct page * page;
+	void *addr;
+
+	page = get_object_page(object);
+
+	if (!page || sc != page->slab)
+		return 0;
+
+	addr = page_address(page);
+	if (object < addr || object >= addr + s->objects * s->size)
+		return 0;
+
+	if ((object - addr) & s->size)
+		return 0;
+
+	return 1;
+}
+
+/*
+ * Determine the size of a slab object
+ */
+static unsigned long slab_object_size(struct slab_cache *sc,
+						const void *object)
+{
+	struct page *page;
+	struct slab_cache *s;
+
+
+	page = get_object_page(object);
+	if (page) {
+		s = page->slab;
+		BUG_ON(sc && s != sc);
+		if (s)
+			return sc->size;
+	}
+	BUG();
+	return 0;	/* Satisfy compiler */
+}
+
+/*
+ * Move slab objects in a given slab by calling the move_objects function.
+ *
+ * Must be called with the slab lock held but will drop and reacquire the
+ * slab lock.
+ */
+static int move_slab_objects(struct slab *s, struct page *page,
+			 int (*move_objects)(struct slab_cache *, void *))
+{
+	int unfreeable = 0;
+	void *addr = page_address(page);
+
+	while (page->inuse - unfreeable > 0) {
+		void *p;
+
+		for (p = addr; p < addr + s->objects; p+= s->size) {
+			if (!on_freelist(s, page, p)) {
+				/*
+				 * Drop the lock here to allow the
+				 * move_object function to do things
+				 * with the slab_cache and maybe this
+				 * page.
+				 */
+				slab_unlock(page);
+				local_irq_enable();
+				if (move_objects((struct slab_cache *)s, p))
+					slab_free(&s->sc, p);
+				else
+					unfreeable++;
+				local_irq_disable();
+				slab_lock(page);
+			}
+		}
+	}
+	return unfreeable;
+}
+
+/*
+ * Shrinking drops the active per cpu slabs and also reaps all empty
+ * slabs off the partial list. Returns the number of slabs freed.
+ *
+ * The move_object function will be called for each objects in partially
+ * allocated slabs. move_object() needs to perform a new allocation for
+ * the object and move the contents of the object to the new location.
+ *
+ * If move_object() returns 1 for success then the object is going to be
+ * removed. If 0 then the object cannot be freed at all. As a result the
+ * slab containing the object will also not be freeable.
+ *
+ * Returns the number of slabs freed.
+ */
+static int slab_shrink(struct slab_cache *sc,
+			int (*move_object)(struct slab_cache *, void *))
+{
+	struct slab *s = (void *)sc;
+	unsigned long flags;
+	int slabs_freed = 0;
+	int i;
+
+	drain_all(s);
+
+	local_irq_save(flags);
+	for(i = 0; s->nr_partial > 1 && i < s->nr_partial - 1; i++ ) {
+		struct page * page;
+
+		page = get_partial(s, -1);
+		if (!page)
+			break;
+
+		/*
+		 * Pin page so that slab_free will not free even if we
+		 * drop the slab lock.
+		 */
+		__SetPageActive(page);
+
+		if (page->inuse < s->objects && move_object)
+			if (move_slab_objects(s,
+					page, move_object) == 0)
+				slabs_freed++;
+
+		/*
+		 * This will put the slab on the front of the partial
+		 * list, the used list or free it.
+		 */
+		__ClearPageActive(page);
+		putback_slab(s, page);
+	}
+	local_irq_restore(flags);
+	return slabs_freed;
+
+}
+
+static struct slab_cache *slab_dup(struct slab_cache *sc)
+{
+	struct slab *s = (void *)sc;
+
+	atomic_inc(&s->refcount);
+	return &s->sc;
+}
+
+static int free_list(struct slab *s, struct list_head *list)
+{
+	int slabs_inuse = 0;
+	unsigned long flags;
+	struct page *page, *h;
+
+	spin_lock_irqsave(&s->list_lock, flags);
+	list_for_each_entry_safe(page, h, list, lru)
+		if (!page->inuse) {
+			list_del(&s->partial);
+			discard_slab(s, page);
+		} else
+			slabs_inuse++;
+	spin_unlock_irqrestore(&s->list_lock, flags);
+	return slabs_inuse;
+}
+
+static int slab_destroy(struct slab_cache *sc)
+{
+	struct slab *s = (void *)sc;
+
+	if (!atomic_dec_and_test(&s->refcount))
+		return 0;
+
+	drain_all(s);
+	free_list(s, &s->partial);
+
+	if (atomic_long_read(&s->nr_slabs))
+		return 1;
+
+	/* Just to make sure that no one uses this again */
+	s->size = 0;
+	return 0;
+}
+
+static unsigned long count_objects(struct slab *s, struct list_head *list)
+{
+	int count = 0;
+	struct page *page;
+	unsigned long flags;
+
+	spin_lock_irqsave(&s->list_lock, flags);
+	list_for_each_entry(page, list, lru)
+		count += page->inuse;
+	spin_unlock_irqrestore(&s->list_lock, flags);
+	return count;
+}
+
+static unsigned long slab_objects(struct slab_cache *sc,
+	unsigned long *p_total, unsigned long *p_active,
+	unsigned long *p_partial)
+{
+	struct slab *s = (void *)sc;
+	int partial = count_objects(s, &s->partial);
+	int nr_slabs = atomic_read(&s->nr_slabs);
+	int active = 0;		/* Active slabs */
+	int nr_active = 0;	/* Objects in active slabs */
+	int cpu;
+
+	for_each_possible_cpu(cpu) {
+		struct page *page = s->active[cpu];
+
+		if (page) {
+			nr_active++;
+			active += page->inuse;
+		}
+	}
+
+	if (p_partial)
+		*p_partial = s->nr_partial;
+
+	if (p_active)
+		*p_active = nr_active;
+
+	if (p_total)
+		*p_total = nr_slabs;
+
+	return partial + active +
+		(nr_slabs - s->nr_partial - nr_active) * s->objects;
+}
+
+const struct slab_allocator slabifier_allocator = {
+	.name = "Slabifier",
+	.create = slab_create,
+	.alloc = slab_alloc,
+	.alloc_node = slab_alloc_node,
+	.free = slab_free,
+	.valid_pointer = slab_pointer_valid,
+	.object_size = slab_object_size,
+	.get_objects = slab_objects,
+	.shrink = slab_shrink,
+	.dup = slab_dup,
+	.destroy = slab_destroy,
+	.destructor = null_slab_allocator_destructor,
+};
+EXPORT_SYMBOL(slabifier_allocator);
mm/Makefile +1 -1
Index: linux-2.6.18-rc5-mm1/mm/Makefile
===================================================================
@@ -28,4 +28,4 @@ obj-$(CONFIG_MEMORY_HOTPLUG) += memory_h
 obj-$(CONFIG_FS_XIP) += filemap_xip.o
 obj-$(CONFIG_MIGRATION) += migrate.o
 obj-$(CONFIG_SMP) += allocpercpu.o
-obj-$(CONFIG_MODULAR_SLAB) += allocator.o
+obj-$(CONFIG_MODULAR_SLAB) += allocator.o slabifier.o
include/linux/mm.h +15 -2
Index: linux-2.6.18-rc5-mm1/include/linux/mm.h
===================================================================
@@ -226,10 +226,16 @@ struct page {
 	unsigned long flags;		/* Atomic flags, some possibly
 					 * updated asynchronously */
 	atomic_t _count;		/* Usage count, see below. */
-	atomic_t _mapcount;		/* Count of ptes mapped in mms,
+	union {
+		atomic_t _mapcount;	/* Count of ptes mapped in mms,
 					 * to show when page is mapped
 					 * & limit reverse map searches.
 					 */
+		struct {	/* Slabifier */
+			short unsigned int inuse;
+			short unsigned int offset;
+		};
+	};
 	union {
 	    struct {
 		unsigned long private;		/* Mapping-private opaque data:
@@ -250,8 +256,15 @@ struct page {
 #if NR_CPUS >= CONFIG_SPLIT_PTLOCK_CPUS
 	    spinlock_t ptl;
 #endif
+	    struct {			/* Slabifier */
+		struct page *first_page;	/* Compound pages */
+		struct slab_cache *slab;	/* Pointer to slab */
+	    };
+	};
+	union {
+		pgoff_t index;		/* Our offset within mapping. */
+		void *freelist;		/* Slabifier: free object */
 	};
-	pgoff_t index;			/* Our offset within mapping. */
 	struct list_head lru;		/* Pageout list, eg. active_list
 					 * protected by zone->lru_lock !
 					 */
include/linux/page-flags.h +8 -1
Index: linux-2.6.18-rc5-mm1/include/linux/page-flags.h
===================================================================
@@ -92,7 +92,7 @@
 #define PG_buddy		19	/* Page is free, on buddy lists */
 
 #define PG_readahead		20	/* Reminder to do readahead */
-
+#define PG_slabsingle		21	/* Slab with a single object */
 
 #if (BITS_PER_LONG > 32)
 /*
@@ -126,6 +126,8 @@
 #define PageReferenced(page)	test_bit(PG_referenced, &(page)->flags)
 #define SetPageReferenced(page)	set_bit(PG_referenced, &(page)->flags)
 #define ClearPageReferenced(page)	clear_bit(PG_referenced, &(page)->flags)
+#define __SetPageReferenced(page) __set_bit(PG_referenced, &(page)->flags)
+#define __ClearPageReferenced(page) __clear_bit(PG_referenced, &(page)->flags)
 #define TestClearPageReferenced(page) test_and_clear_bit(PG_referenced, &(page)->flags)
 
 #define PageUptodate(page)	test_bit(PG_uptodate, &(page)->flags)
@@ -155,6 +157,7 @@
 
 #define PageActive(page)	test_bit(PG_active, &(page)->flags)
 #define SetPageActive(page)	set_bit(PG_active, &(page)->flags)
+#define __SetPageActive(page)	__set_bit(PG_active, &(page)->flags)
 #define ClearPageActive(page)	clear_bit(PG_active, &(page)->flags)
 #define __ClearPageActive(page)	__clear_bit(PG_active, &(page)->flags)
 
@@ -254,6 +257,10 @@
 #define SetPageReadahead(page)	set_bit(PG_readahead, &(page)->flags)
 #define TestClearPageReadahead(page) test_and_clear_bit(PG_readahead, &(page)->flags)
 
+#define PageSlabsingle(page)	test_bit(PG_slabsingle, &(page)->flags)
+#define __SetPageSlabsingle(page) __set_bit(PG_slabsingle, &(page)->flags)
+#define __ClearPageSlabsingle(page) __clear_bit(PG_slabsingle, &(page)->flags)
+
 struct page;	/* forward declaration */
 
 int test_clear_page_dirty(struct page *page);

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#3 lore raw

PATCH 3/5 /proc/slabinfo display

In reply to Christoph Lameter

Generic Slab statistics module

A statistic module for generic slab allocator framework.

The creator of a cache must register the slab cache with

register_slab()

in order for something to show up in slabinfo.

Here is a sample of slabinfo output:

slabinfo - version: 3.0

# name            <objects> <objsize> <num_slabs> <partial_slabs> <active_slabs> <order> <allocator>
nfs_direct_cache           0     136       0       0       0  0 reclaimable:page_allocator
nfs_write_data            36     896       2       0       0  0 unreclaimable:page_allocator
nfs_read_data             21     768       2       1       0  0 unreclaimable:page_allocator
nfs_inode_cache           17    1032       3       3       0  0 ctor_dtor:reclaimable:page_allocator
rpc_tasks                  0     384       1       1       0  0 unreclaimable:page_allocator
rpc_inode_cache            0     896       1       1       0  0 ctor_dtor:reclaimable:page_allocator
ip6_dst_cache              0     384       1       1       0  0 unreclaimable:page_allocator
TCPv6                      1    1792       2       2       0  0 unreclaimable:page_allocator
UNIX                     112     768      11       7       0  0 unreclaimable:page_allocator
dm_tio                     0      24       0       0       0  0 unreclaimable:page_allocator
dm_io                      0      40       0       0       0  0 unreclaimable:page_allocator
kmalloc                    0      64       0       0       0  0 dma:unreclaimable:page_allocator
cfq_ioc_pool               0     160       0       0       0  0 unreclaimable:page_allocator
cfq_pool                   0     160       0       0       0  0 unreclaimable:page_allocator
mqueue_inode_cache         0     896       1       1       0  0 ctor_dtor:unreclaimable:page_allocator
xfs_chashlist            822      40       9       9       0  0 unreclaimable:page_allocator
xfs_ili                  183     192       9       8       1  0 unreclaimable:page_allocator
xfs_inode               7890     640     319       6       1  0 reclaimable:page_allocator
xfs_efi_item               0     352       0       0       0  0 unreclaimable:page_allocator
xfs_efd_item               0     360       0       0       0  0 unreclaimable:page_allocator
xfs_buf_item               4     184       1       0       1  0 unreclaimable:page_allocator
xfs_acl                    0     304       0       0       0  0 unreclaimable:page_allocator
xfs_dabuf                  0      24       1       0       1  0 unreclaimable:page_allocator
xfs_da_state               0     488       0       0       0  0 unreclaimable:page_allocator
xfs_trans                  1     832       1       0       1  0 unreclaimable:page_allocator
xfs_btree_cur              0     192       1       0       1  0 unreclaimable:page_allocator
xfs_bmap_free_item         0      24       0       0       0  0 unreclaimable:page_allocator
xfs_ioend                128     160       2       0       1  0 unreclaimable:page_allocator
xfs_vnode               7891     768     380       5       1  0 ctor_dtor:reclaimable:page_allocator
isofs_inode_cache          0     656       0       0       0  0 ctor_dtor:reclaimable:page_allocator
fat_inode_cache            0     688       1       1       0  0 ctor_dtor:reclaimable:page_allocator
fat_cache                  0      40       0       0       0  0 ctor_dtor:reclaimable:page_allocator
hugetlbfs_inode_cache      0     624       1       1       0  0 ctor_dtor:unreclaimable:page_allocator
ext2_inode_cache           0     776       0       0       0  0 ctor_dtor:reclaimable:page_allocator
ext2_xattr                 0      88       0       0       0  0 reclaimable:page_allocator
journal_handle             0      24       0       0       0  0 unreclaimable:page_allocator
journal_head               0      96       0       0       0  0 unreclaimable:page_allocator
ext3_inode_cache           0     824       0       0       0  0 ctor_dtor:reclaimable:page_allocator
ext3_xattr                 0      88       0       0       0  0 reclaimable:page_allocator
reiser_inode_cache         0     736       0       0       0  0 ctor_dtor:reclaimable:page_allocator
dnotify_cache              0      40       0       0       0  0 unreclaimable:page_allocator
dquot                      0     256       0       0       0  0 reclaimable:page_allocator
eventpoll_pwq              0      72       1       1       0  0 unreclaimable:page_allocator
inotify_event_cache        0      40       0       0       0  0 unreclaimable:page_allocator
inotify_watch_cache        0      72       1       1       0  0 unreclaimable:page_allocator
kioctx                     0     384       0       0       0  0 unreclaimable:page_allocator
fasync_cache               0      24       0       0       0  0 unreclaimable:page_allocator
shmem_inode_cache        794     816      45      12       0  0 ctor_dtor:unreclaimable:page_allocator
posix_timers_cache         0     136       0       0       0  0 unreclaimable:page_allocator
partial_page_cache         0      48       0       0       0  0 unreclaimable:page_allocator
xfrm_dst_cache             0     384       0       0       0  0 unreclaimable:page_allocator
ip_dst_cache              21     384       2       2       0  0 unreclaimable:page_allocator
RAW                        0     896       1       1       0  0 unreclaimable:page_allocator
UDP                        3     896       3       2       1  0 unreclaimable:page_allocator
TCP                       12    1664       4       4       0  0 unreclaimable:page_allocator
scsi_io_context            0     112       0       0       0  0 unreclaimable:page_allocator
blkdev_ioc                26      56       7       7       0  0 unreclaimable:page_allocator
blkdev_queue              24    1616       4       2       0  0 unreclaimable:page_allocator
blkdev_requests           12     280       2       0       2  0 unreclaimable:page_allocator
sock_inode_cache         167     768      12       6       1  0 ctor_dtor:reclaimable:page_allocator
file_lock_cache            1     184       2       2       0  0 ctor_dtor:unreclaimable:page_allocator
Acpi-Parse                 0      40       0       0       0  0 unreclaimable:page_allocator
Acpi-State                 0      80       0       0       0  0 unreclaimable:page_allocator
proc_inode_cache         696     640      36      16       1  0 ctor_dtor:reclaimable:page_allocator
sigqueue                   0     160       4       0       4  0 unreclaimable:page_allocator
radix_tree_node         2068     560      75       5       0  0 ctor_dtor:unreclaimable:page_allocator
bdev_cache                42     896       5       4       0  0 ctor_dtor:reclaimable:page_allocator
sysfs_dir_cache         4283      80      24       4       0  0 unreclaimable:page_allocator
inode_cache             2571     608     103       8       1  0 ctor_dtor:reclaimable:page_allocator
dentry_cache           13014     200     166       7       3  0 reclaimable:page_allocator
idr_layer_cache           76     536       4       2       0  0 ctor_dtor:unreclaimable:page_allocator
buffer_head             4417     104      33       9       0  0 ctor_dtor:reclaimable:page_allocator
vm_area_struct          1503     176      24      19       3  0 unreclaimable:page_allocator
files_cache               47     768       7       6       1  0 unreclaimable:page_allocator
signal_cache             136     640      10       6       1  0 unreclaimable:page_allocator
sighand_cache            136    1664      19       6       1  0 ctor_dtor:rcu:unreclaimable:page_allocator
anon_vma                 264      32       9       8       1  0 ctor_dtor:rcu:unreclaimable:page_allocator
shared_policy_node         0      48       0       0       0  0 unreclaimable:page_allocator
numa_policy               85     264       4       3       0  0 unreclaimable:page_allocator
kmalloc                    0  262144       0       0       0  4 unreclaimable:page_allocator
kmalloc                    2  131072       2       0       0  3 unreclaimable:page_allocator
kmalloc                    1   65536       1       0       0  2 unreclaimable:page_allocator
kmalloc                   10   32768      10       0       0  1 unreclaimable:page_allocator
kmalloc                   93   16384      93       0       0  0 unreclaimable:page_allocator
kmalloc                   98    8192      49       0       0  0 unreclaimable:page_allocator
kmalloc                   99    4096      31       8       4  0 unreclaimable:page_allocator
kmalloc                  345    2048      47      14       2  0 unreclaimable:page_allocator
kmalloc                  228    1024      21      12       2  0 unreclaimable:page_allocator
kmalloc                  183     512      14       9       3  0 unreclaimable:page_allocator
kmalloc                 3892     256      78      31       3  0 unreclaimable:page_allocator
kmalloc                 1244     128      18       9       4  0 unreclaimable:page_allocator
kmalloc                 1619      64      12       8       1  0 unreclaimable:page_allocator
kmalloc                  121      32       8       5       3  0 unreclaimable:page_allocator
kmalloc                 1644      16       5       4       0  0 unreclaimable:page_allocator
kmalloc                  128       8       4       4       0  0 unreclaimable:page_allocator
fs/proc/proc_misc.c +8 -4
Index: linux-2.6.18-rc5-mm1/fs/proc/proc_misc.c
===================================================================
@@ -399,9 +399,11 @@ static struct file_operations proc_modul
 };
 #endif
 
-#ifdef CONFIG_SLAB
+#if defined(CONFIG_SLAB) || defined(CONFIG_MODULAR_SLAB)
 extern struct seq_operations slabinfo_op;
+#ifdef CONFIG_SLAB
 extern ssize_t slabinfo_write(struct file *, const char __user *, size_t, loff_t *);
+#endif
 static int slabinfo_open(struct inode *inode, struct file *file)
 {
 	return seq_open(file, &slabinfo_op);
@@ -409,12 +411,14 @@ static int slabinfo_open(struct inode *i
 static struct file_operations proc_slabinfo_operations = {
 	.open		= slabinfo_open,
 	.read		= seq_read,
+#ifdef CONFIG_SLAB
 	.write		= slabinfo_write,
+#endif
 	.llseek		= seq_lseek,
 	.release	= seq_release,
 };
 
-#ifdef CONFIG_DEBUG_SLAB_LEAK
+#if defined(CONFIG_DEBUG_SLAB_LEAK) && defined(CONFIG_SLAB)
 extern struct seq_operations slabstats_op;
 static int slabstats_open(struct inode *inode, struct file *file)
 {
@@ -787,9 +791,9 @@ void __init proc_misc_init(void)
 #endif
 	create_seq_entry("stat", 0, &proc_stat_operations);
 	create_seq_entry("interrupts", 0, &proc_interrupts_operations);
-#ifdef CONFIG_SLAB
+#if defined(CONFIG_SLAB) || defined(CONFIG_MODULAR_SLAB)
 	create_seq_entry("slabinfo",S_IWUSR|S_IRUGO,&proc_slabinfo_operations);
-#ifdef CONFIG_DEBUG_SLAB_LEAK
+#if defined(CONFIG_DEBUG_SLAB_LEAK) && defined(CONFIG_SLAB)
 	create_seq_entry("slab_allocators", 0 ,&proc_slabstats_operations);
 #endif
 #endif
mm/Makefile +1 -1
Index: linux-2.6.18-rc5-mm1/mm/Makefile
===================================================================
@@ -28,4 +28,4 @@ obj-$(CONFIG_MEMORY_HOTPLUG) += memory_h
 obj-$(CONFIG_FS_XIP) += filemap_xip.o
 obj-$(CONFIG_MIGRATION) += migrate.o
 obj-$(CONFIG_SMP) += allocpercpu.o
-obj-$(CONFIG_MODULAR_SLAB) += allocator.o slabifier.o
+obj-$(CONFIG_MODULAR_SLAB) += allocator.o slabifier.o slabstat.o
mm/slabstat.c new file +96 -0

96 lines not on this page · raw message

include/linux/slabstat.h new file +9 -0
Index: linux-2.6.18-rc5-mm1/include/linux/slabstat.h
===================================================================
@@ -0,0 +1,9 @@
+#ifndef _LINUX_SLABSTAT_H
+#define _LINUX_SLABSTAT_H
+#include <linux/allocator.h>
+
+void register_slab(struct slab_cache *s);
+void unregister_slab(struct slab_cache *s);
+
+#endif /* _LINUX_SLABSTAT_H */
+

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#4 lore raw

PATCH 4/5 Kmalloc subsystem

In reply to Christoph Lameter

A generic kmalloc layer for the modular slab

Regular kmalloc allocations are optimized. DMA kmalloc slabs are created on demand.

Also re exports the kmalloc array as a new slab_allocator that can be used to tie into the kmalloc array (the slabulator uses that to avoid creating new slabs that are compatible with generic kmalloc caches).

include/linux/kmalloc.h new file +136 -0
Index: linux-2.6.18-rc5-mm1/include/linux/kmalloc.h
===================================================================
@@ -0,0 +1,136 @@
+#ifndef _LINUX_KMALLOC_H
+#define _LINUX_KMALLOC_H
+/*
+ * In kernel dynamic memory allocator.
+ *
+ * (C) 2006 Silicon Graphics, Inc,
+ * 		Christoph Lameter <[email protected]>
+ */
+
+#include <linux/allocator.h>
+#include <linux/config.h>
+#include <linux/types.h>
+
+#ifndef KMALLOC_ALLOCATOR
+#define KMALLOC_ALLOCATOR slabifier_allocator
+#endif
+
+#define KMALLOC_SHIFT_LOW 3
+
+#define KMALLOC_SHIFT_HIGH 18
+
+#if L1_CACHE_BYTES <= 64
+#define KMALLOC_EXTRAS 2
+#define KMALLOC_EXTRA
+#else
+#define KMALLOC_EXTRAS 0
+#endif
+
+#define KMALLOC_NR_CACHES (KMALLOC_SHIFT_HIGH - KMALLOC_SHIFT_LOW \
+			 + 1 + KMALLOC_EXTRAS)
+/*
+ * We keep the general caches in an array of slab caches that are used for
+ * 2^x bytes of allocations. For each size we generate a DMA and a
+ * non DMA cache (DMA simply means memory for legacy I/O. The regular
+ * caches can be used for devices that can DMA to all of memory).
+ */
+extern struct slab_control kmalloc_caches[KMALLOC_NR_CACHES];
+
+/*
+ * Sorry that the following has to be that ugly but GCC has trouble
+ * with constant propagation and loops.
+ */
+static inline int kmalloc_index(int size)
+{
+	if (size <=    8) return 3;
+	if (size <=   16) return 4;
+	if (size <=   32) return 5;
+	if (size <=   64) return 6;
+#ifdef KMALLOC_EXTRA
+	if (size <=   96) return KMALLOC_SHIFT_HIGH + 1;
+#endif
+	if (size <=  128) return 7;
+#ifdef KMALLOC_EXTRA
+	if (size <=  192) return KMALLOC_SHIFT_HIGH + 2;
+#endif
+	if (size <=  256) return 8;
+	if (size <=  512) return 9;
+	if (size <= 1024) return 10;
+	if (size <= 2048) return 11;
+	if (size <= 4096) return 12;
+	if (size <=   8 * 1024) return 13;
+	if (size <=  16 * 1024) return 14;
+	if (size <=  32 * 1024) return 15;
+	if (size <=  64 * 1024) return 16;
+	if (size <= 128 * 1024) return 17;
+	if (size <= 256 * 1024) return 18;
+	return -1;
+}
+
+/*
+ * Find the slab cache for a given combination of allocation flags and size.
+ *
+ * This ought to end up with a global pointer to the right cache
+ * in kmalloc_caches.
+ */
+static inline struct slab_cache *kmalloc_slab(size_t size)
+{
+	int index = kmalloc_index(size) - KMALLOC_SHIFT_LOW;
+
+	if (index < 0) {
+		/*
+		 * Generate a link failure. Would be great if we could
+		 * do something to stop the compile here.
+		 */
+		extern void __kmalloc_size_too_large(void);
+		__kmalloc_size_too_large();
+	}
+	return &kmalloc_caches[index].sc;
+}
+
+extern void *__kmalloc(size_t, gfp_t);
+#define ____kmalloc __kmalloc
+
+static inline void *kmalloc(size_t size, gfp_t flags)
+{
+	if (__builtin_constant_p(size) && !(flags & __GFP_DMA)) {
+		struct slab_cache *s = kmalloc_slab(size);
+
+		return KMALLOC_ALLOCATOR.alloc(s, flags);
+	} else
+		return __kmalloc(size, flags);
+}
+
+#ifdef CONFIG_NUMA
+extern void *__kmalloc_node(size_t, gfp_t, int);
+static inline void *kmalloc_node(size_t size, gfp_t flags, int node)
+{
+	if (__builtin_constant_p(size) && !(flags & __GFP_DMA)) {
+		struct slab_cache *s = kmalloc_slab(size);
+
+		return KMALLOC_ALLOCATOR.alloc_node(s, flags, node);
+	} else
+		return __kmalloc_node(size, flags, node);
+}
+#else
+#define kmalloc_node(__size, __flags, __node) kmalloc((__size), (__flags))
+#endif
+
+/* Free an object */
+static inline void kfree(const void *x)
+{
+	return KMALLOC_ALLOCATOR.free(NULL, x);
+}
+
+/* Allocate and zero the specified number of bytes */
+extern void *kzalloc(size_t, gfp_t);
+
+/* Figure out what size the chunk is */
+extern size_t ksize(const void *);
+
+extern struct page_allocator *reclaimable_allocator;
+extern struct page_allocator *unreclaimable_allocator;
+
+extern int slab_min_order;
+
+#endif	/* _LINUX_KMALLOC_H */
mm/kmalloc.c new file +226 -0

226 lines not on this page · raw message

mm/Makefile +1 -1
Index: linux-2.6.18-rc5-mm1/mm/Makefile
===================================================================
@@ -28,4 +28,4 @@ obj-$(CONFIG_MEMORY_HOTPLUG) += memory_h
 obj-$(CONFIG_FS_XIP) += filemap_xip.o
 obj-$(CONFIG_MIGRATION) += migrate.o
 obj-$(CONFIG_SMP) += allocpercpu.o
-obj-$(CONFIG_MODULAR_SLAB) += allocator.o slabifier.o slabstat.o
+obj-$(CONFIG_MODULAR_SLAB) += allocator.o slabifier.o slabstat.o kmalloc.o
include/asm-i386/page.h +1 -0
Index: linux-2.6.18-rc5-mm1/include/asm-i386/page.h
===================================================================
@@ -37,6 +37,7 @@
 
 #define alloc_zeroed_user_highpage(vma, vaddr) alloc_page_vma(GFP_HIGHUSER | __GFP_ZERO, vma, vaddr)
 #define __HAVE_ARCH_ALLOC_ZEROED_USER_HIGHPAGE
+#define ARCH_NEEDS_SMALL_SLABS
 
 /*
  * These are used to make use of C type-checking..

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#5 lore raw

PATCH 5/5 Slabulator: Emulate the existing Slab Layer

In reply to Christoph Lameter

The slab emulation layer.

This provides a layer that implements the existing slab API. We try to keep the definitions that we copy from slab.h to an absolute minimum. If things break then more (useless) definitions from slab.h may be needed.

We put a hook into slab.h to redirect includes for slab.h to slabulator.h.

The slabulator also contains the remnants of the slab reaper since it is used by the page allocator in the CONFIG_NUMA case. The slabifier does not need this anymore since it is not object cache based.

mm/slabulator.c new file +299 -0
Index: linux-2.6.18-rc5-mm1/mm/slabulator.c
===================================================================
@@ -0,0 +1,299 @@
+/*
+ * Slabulator = Emulate the Slab API.
+ *
+ * (C) 2006 Silicon Graphics, Inc. Christoph Lameter <[email protected]>
+ *
+ */
+#include <linux/mm.h>
+#include <linux/kmalloc.h>
+#include <linux/module.h>
+#include <linux/allocator.h>
+#include <linux/bitops.h>
+#include <linux/slabulator.h>
+#include <linux/slabstat.h>
+
+#define SLAB_MAX_ORDER 4
+
+#define SLABULATOR_MERGE
+
+#ifndef ARCH_SLAB_MINALIGN
+#define ARCH_SLAB_MINALIGN sizeof(void *)
+#endif
+
+static int calculate_order(int size)
+{
+	int order;
+	int rem;
+
+	if ((size & (size -1)) == 0) {
+		/*
+		 * We can use the page allocator if the requested size
+		 * is compatible with the page sizes supported.
+		 */
+		int order = fls(size) -1 - PAGE_SHIFT;
+
+		if (order >= 0)
+			return 0;
+	}
+
+	for(order = max(slab_min_order, fls(size - 1) - PAGE_SHIFT);
+			order < MAX_ORDER; order++) {
+		unsigned long slab_size = PAGE_SIZE << order;
+
+		if (slab_size < size)
+			continue;
+
+		rem = slab_size % size;
+
+		if (rem * 8 <= PAGE_SIZE << order)
+			break;
+
+	}
+	if (order >= MAX_ORDER)
+		return -E2BIG;
+	return order;
+}
+
+/*
+ * We can actually operate slabs any time after the page allocator is up.
+ * slab_is_available() merely means that the kmalloc array is available.
+ *
+ * However, be aware that deriving allocators depends on kmalloc being
+ * functional.
+ */
+int slabulator_up = 0;
+
+int slab_is_available(void) {
+	return slabulator_up;
+}
+
+void kmem_cache_init(void)
+{
+	extern void kmalloc_init(void);
+
+	kmalloc_init();
+	slabulator_up = 1;
+}
+
+struct slab_cache *kmem_cache_create(const char *name, size_t size,
+		size_t align, unsigned long flags,
+		void (*ctor)(void *, struct slab_cache *, unsigned long),
+		void (*dtor)(void *, struct slab_cache *, unsigned long))
+{
+	const struct page_allocator *a;
+	struct slab_cache s;
+	struct slab_cache *rs;
+	struct slab_control *x;
+	int one_object_slab;
+
+	s.offset = 0;
+	s.align = max(ARCH_SLAB_MINALIGN, ALIGN(align, sizeof(void *)));
+
+	if (flags & (SLAB_MUST_HWCACHE_ALIGN|SLAB_HWCACHE_ALIGN))
+		s.align = L1_CACHE_BYTES;
+
+	s.inuse = size;
+	s.objsize = size;
+	s.size = ALIGN(size, s.align);
+
+	/* Pick the right allocator for our purposes */
+	if (flags & SLAB_RECLAIM_ACCOUNT)
+		a = reclaimable_allocator;
+	else
+		a = unreclaimable_allocator;
+
+	if (flags & SLAB_CACHE_DMA)
+		a = dmaify_page_allocator(a);
+
+	if (flags & SLAB_DESTROY_BY_RCU)
+		a = rcuify_page_allocator(a);
+
+	one_object_slab = s.size > ((PAGE_SIZE / 2)  << calculate_order(s.size));
+
+	if (!one_object_slab && ((flags & SLAB_DESTROY_BY_RCU) || ctor || dtor)) {
+		/*
+		 * For RCU processing and constructors / destructors:
+		 * The object must remain intact even if it is free.
+		 * The free pointer would hurt us there.
+		 * Relocate the free object pointer out of
+		 * the space used by the object.
+		 *
+		 * Slabs with a single object do not need this since
+		 * those do not have to deal with free pointers.
+		 */
+		s.offset = s.size - sizeof(void *);
+		if (s.offset < s.objsize) {
+			/*
+			 * Would overlap the object. We need to waste some
+			 * more space to make the object safe from the
+			 * free pointer.
+			 */
+			s.offset = s.size;
+			s.size += s.align;
+		}
+		s.inuse = s.size;
+	}
+
+	s.order = calculate_order(s.size);
+
+	if (s.order < 0)
+		goto error;
+
+	s.name = name;
+	s.node = -1;
+
+	x = kmalloc(sizeof(struct slab_control), GFP_KERNEL);
+
+	if (!x)
+		return NULL;
+	s.page_alloc = a;
+	s.slab_alloc = &SLABULATOR_ALLOCATOR;
+#ifdef SLABULATOR_MERGE
+	/*
+	 * This works but is this really something we want?
+	 */
+	if (((s.size & (s.size - 1))==0) && !ctor && !dtor &&
+		   !(flags & (SLAB_DESTROY_BY_RCU|SLAB_RECLAIM_ACCOUNT))) {
+
+		printk(KERN_INFO "Merging slab_cache %s size %d into"
+			" kmalloc array\n", name, s.size);
+		rs = kmalloc_slab_allocator.create(x, &s);
+		kfree(x);
+		x = NULL;
+	} else
+#endif
+	rs = SLABULATOR_ALLOCATOR.create(x, &s);
+	if (!rs)
+		goto error;
+
+	/*
+	 * Now deal with constuctors and destructors. We need to know the
+	 * slab_cache address in order to be able to pass the slab_cache
+	 * address down the chain.
+	 */
+	if (ctor || dtor)
+		rs->page_alloc =
+			ctor_and_dtor_for_page_allocator(rs->page_alloc,
+				rs->size, rs,
+				(void *)ctor, (void *)dtor);
+
+	if (x)
+		register_slab(rs);
+	return rs;
+
+error:
+	a->destructor((struct page_allocator *)a);
+	if (flags & SLAB_PANIC)
+		panic("Cannot create slab %s size=%d realsize=%d "
+			"order=%d offset=%d flags=%lx\n",
+			s.name, size, s.size, s.order, s.offset, flags);
+
+
+	return NULL;
+}
+EXPORT_SYMBOL(kmem_cache_create);
+
+int kmem_cache_destroy(struct slab_cache *s)
+{
+	SLABULATOR_ALLOCATOR.destroy(s);
+	unregister_slab(s);
+	kfree(s);
+	return 0;
+}
+EXPORT_SYMBOL(kmem_cache_destroy);
+
+void *kmem_cache_zalloc(struct slab_cache *s, gfp_t flags)
+{
+	void *x;
+
+	x = kmem_cache_alloc(s, flags);
+	if (x)
+		memset(x, 0, s->objsize);
+	return x;
+}
+
+/*
+ * Generic reaper (the slabifier has its own way of reaping)
+ */
+#ifdef CONFIG_NUMA
+/*
+ * Special reaping functions for NUMA systems called from cache_reap().
+ */
+static DEFINE_PER_CPU(unsigned long, reap_node);
+
+static void init_reap_node(int cpu)
+{
+	int node;
+
+	node = next_node(cpu_to_node(cpu), node_online_map);
+	if (node == MAX_NUMNODES)
+		node = first_node(node_online_map);
+
+	__get_cpu_var(reap_node) = node;
+}
+
+static void next_reap_node(void)
+{
+	int node = __get_cpu_var(reap_node);
+
+	/*
+	 * Also drain per cpu pages on remote zones
+	 */
+	if (node != numa_node_id())
+		drain_node_pages(node);
+
+	node = next_node(node, node_online_map);
+	if (unlikely(node >= MAX_NUMNODES))
+		node = first_node(node_online_map);
+	__get_cpu_var(reap_node) = node;
+}
+#else
+#define init_reap_node(cpu) do { } while (0)
+#define next_reap_node(void) do { } while (0)
+#endif
+
+#define REAPTIMEOUT_CPUC	(2*HZ)
+
+#ifdef CONFIG_SMP
+static DEFINE_PER_CPU(struct work_struct, reap_work);
+
+static void cache_reap(void *unused)
+{
+	next_reap_node();
+	refresh_cpu_vm_stats(smp_processor_id());
+
+	schedule_delayed_work(&__get_cpu_var(reap_work),
+				      REAPTIMEOUT_CPUC);
+}
+
+static void __devinit start_cpu_timer(int cpu)
+{
+	struct work_struct *reap_work = &per_cpu(reap_work, cpu);
+
+	/*
+	 * When this gets called from do_initcalls via cpucache_init(),
+	 * init_workqueues() has already run, so keventd will be setup
+	 * at that time.
+	 */
+	if (keventd_up() && reap_work->func == NULL) {
+		init_reap_node(cpu);
+		INIT_WORK(reap_work, cache_reap, NULL);
+		schedule_delayed_work_on(cpu, reap_work, HZ + 3 * cpu);
+	}
+}
+
+static int __init cpucache_init(void)
+{
+	int cpu;
+
+	/*
+	 * Register the timers that drain pcp pages and update vm statistics
+	 */
+	for_each_online_cpu(cpu)
+		start_cpu_timer(cpu);
+	return 0;
+}
+__initcall(cpucache_init);
+#endif
+
+
include/linux/slabulator.h new file +123 -0

123 lines not on this page · raw message

mm/Makefile +2 -1
Index: linux-2.6.18-rc5-mm1/mm/Makefile
===================================================================
@@ -28,4 +28,5 @@ obj-$(CONFIG_MEMORY_HOTPLUG) += memory_h
 obj-$(CONFIG_FS_XIP) += filemap_xip.o
 obj-$(CONFIG_MIGRATION) += migrate.o
 obj-$(CONFIG_SMP) += allocpercpu.o
-obj-$(CONFIG_MODULAR_SLAB) += allocator.o slabifier.o slabstat.o kmalloc.o
+obj-$(CONFIG_MODULAR_SLAB) += allocator.o slabifier.o slabstat.o \
+				kmalloc.o slabulator.o
init/Kconfig +21 -11
Index: linux-2.6.18-rc5-mm1/init/Kconfig
===================================================================
@@ -332,6 +332,26 @@ config CC_OPTIMIZE_FOR_SIZE
 
 	  If unsure, say N.
 
+config SLAB
+	default y
+	bool "Traditional SLAB allocator"
+	help
+	  Disabling this allows the use of alternate slab allocators
+	  with less overhead such as SLOB (very simple) or the
+	  use the slabifier with the module allocator framework.
+	  Note that alternate slab allocators may not provide
+	  the complete functionality for slab.
+
+config MODULAR_SLAB
+	default y
+	bool "Use the modular allocator framework"
+	depends on EXPERIMENTAL && !SLAB
+	help
+	 The modular  allocator framework allows the flexible use
+	 of different slab allocators and page allocators for memory
+	 allocation. This will completely replace the existing
+	 slab allocator. Beware this is experimental code.
+
 menuconfig EMBEDDED
 	bool "Configure standard kernel features (for small systems)"
 	help
@@ -370,7 +390,6 @@ config KALLSYMS_EXTRA_PASS
 	   reported.  KALLSYMS_EXTRA_PASS is only a temporary workaround while
 	   you wait for kallsyms to be fixed.
 
-
 config HOTPLUG
 	bool "Support for hot-pluggable devices" if EMBEDDED
 	default y
@@ -445,15 +464,6 @@ config SHMEM
 	  option replaces shmem and tmpfs with the much simpler ramfs code,
 	  which may be appropriate on small systems without swap.
 
-config SLAB
-	default y
-	bool "Use full SLAB allocator" if EMBEDDED
-	help
-	  Disabling this replaces the advanced SLAB allocator and
-	  kmalloc support with the drastically simpler SLOB allocator.
-	  SLOB is more space efficient but does not scale well and is
-	  more susceptible to fragmentation.
-
 config VM_EVENT_COUNTERS
 	default y
 	bool "Enable VM event counters for /proc/vmstat" if EMBEDDED
@@ -475,7 +485,7 @@ config BASE_SMALL
 	default 1 if !BASE_FULL
 
 config SLOB
-	default !SLAB
+	default !SLAB && !MODULAR_SLAB
 	bool
 
 menu "Loadable module support"
include/linux/slab.h +6 -0
Index: linux-2.6.18-rc5-mm1/include/linux/slab.h
===================================================================
@@ -9,6 +9,10 @@
 
 #if	defined(__KERNEL__)
 
+#ifdef CONFIG_MODULAR_SLAB
+#include <linux/slabulator.h>
+#else
+
 typedef struct kmem_cache kmem_cache_t;
 
 #include	<linux/gfp.h>
@@ -291,6 +295,8 @@ extern kmem_cache_t	*fs_cachep;
 extern kmem_cache_t	*sighand_cachep;
 extern kmem_cache_t	*bio_cachep;
 
+#endif /* CONFIG_SLABULATOR */
+
 #endif	/* __KERNEL__ */
 
 #endif	/* _LINUX_SLAB_H */

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#6 lore raw

[MODSLAB] Bypass indirections [for performance testing only]

In reply to Christoph Lameter

Bypass indirections.

This is a patch to bypass indirections so that one can get some statistics on how high the impact of the indirect calls is.

Only use this for testing.

mm/slabifier.c +6 -3
Index: linux-2.6.18-rc5-mm1/mm/slabifier.c
===================================================================
@@ -498,12 +498,13 @@ gotpage:
 	goto redo;
 }
 
-static void *slab_alloc(struct slab_cache *sc, gfp_t gfpflags)
+void *slab_alloc(struct slab_cache *sc, gfp_t gfpflags)
 {
 	return __slab_alloc(sc, gfpflags, -1);
 }
+EXPORT_SYMBOL(slab_alloc);
 
-static void *slab_alloc_node(struct slab_cache *sc, gfp_t gfpflags,
+void *slab_alloc_node(struct slab_cache *sc, gfp_t gfpflags,
 							int node)
 {
 #ifdef CONFIG_NUMA
@@ -512,8 +513,9 @@ static void *slab_alloc_node(struct slab
 	return slab_alloc(sc, gfpflags);
 #endif
 }
+EXPORT_SYMBOL(slab_alloc_node);
 
-static void slab_free(struct slab_cache *sc, const void *x)
+void slab_free(struct slab_cache *sc, const void *x)
 {
 	struct slab *s = (void *)sc;
 	struct page * page;
@@ -617,6 +619,7 @@ dumpret:
 	return;
 #endif
 }
+EXPORT_SYMBOL(slab_free);
 
 /* Figure out on which slab object the object resides */
 static __always_inline struct page *get_object_page(const void *x)
include/linux/kmalloc.h +7 -3
Index: linux-2.6.18-rc5-mm1/include/linux/kmalloc.h
===================================================================
@@ -67,6 +67,10 @@ static inline int kmalloc_index(int size
 	return -1;
 }
 
+extern void *slab_alloc(struct slab_cache *, gfp_t flags);
+extern void *slab_alloc_node(struct slab_cache *, gfp_t, int);
+extern void slab_free(struct slab_cache *, const void *);
+
 /*
  * Find the slab cache for a given combination of allocation flags and size.
  *
@@ -96,7 +100,7 @@ static inline void *kmalloc(size_t size,
 	if (__builtin_constant_p(size) && !(flags & __GFP_DMA)) {
 		struct slab_cache *s = kmalloc_slab(size);
 
-		return KMALLOC_ALLOCATOR.alloc(s, flags);
+		return slab_alloc(s, flags);
 	} else
 		return __kmalloc(size, flags);
 }
@@ -108,7 +112,7 @@ static inline void *kmalloc_node(size_t 
 	if (__builtin_constant_p(size) && !(flags & __GFP_DMA)) {
 		struct slab_cache *s = kmalloc_slab(size);
 
-		return KMALLOC_ALLOCATOR.alloc_node(s, flags, node);
+		return slab_alloc_node(s, flags, node);
 	} else
 		return __kmalloc_node(size, flags, node);
 }
@@ -119,7 +123,7 @@ static inline void *kmalloc_node(size_t 
 /* Free an object */
 static inline void kfree(const void *x)
 {
-	return KMALLOC_ALLOCATOR.free(NULL, x);
+	slab_free(NULL, x);
 }
 
 /* Allocate and zero the specified number of bytes */
mm/kmalloc.c +2 -3
Index: linux-2.6.18-rc5-mm1/mm/kmalloc.c
===================================================================
@@ -124,15 +124,14 @@ static struct slab_cache *get_slab(size_
 
 void *__kmalloc(size_t size, gfp_t flags)
 {
-	return KMALLOC_ALLOCATOR.alloc(get_slab(size, flags), flags);
+	return slab_alloc(get_slab(size, flags), flags);
 }
 EXPORT_SYMBOL(__kmalloc);
 
 #ifdef CONFIG_NUMA
 void *__kmalloc_node(size_t size, gfp_t flags, int node)
 {
-	return KMALLOC_ALLOCATOR.alloc_node(get_slab(size, flags),
-							flags, node);
+	return slab_alloc_node(get_slab(size, flags), flags, node);
 }
 EXPORT_SYMBOL(__kmalloc_node);
 #endif
include/linux/slabulator.h +6 -3
Index: linux-2.6.18-rc5-mm1/include/linux/slabulator.h
===================================================================
@@ -73,20 +73,23 @@ static inline const char *kmem_cache_nam
 
 static inline void *kmem_cache_alloc(struct slab_cache *s, gfp_t flags)
 {
-	return SLABULATOR_ALLOCATOR.alloc(s, flags);
+	return slab_alloc(s, flags);
+	//return SLABULATOR_ALLOCATOR.alloc(s, flags);
 }
 
 static inline void *kmem_cache_alloc_node(struct slab_cache *s,
 					gfp_t flags, int node)
 {
-	return SLABULATOR_ALLOCATOR.alloc_node(s, flags, node);
+	return slab_alloc_node(s, flags, node);
+//	return SLABULATOR_ALLOCATOR.alloc_node(s, flags, node);
 }
 
 extern void *kmem_cache_zalloc(struct slab_cache *s, gfp_t flags);
 
 static inline void kmem_cache_free(struct slab_cache *s, const void *x)
 {
-	SLABULATOR_ALLOCATOR.free(s, x);
+	slab_free(s, x);
+//	SLABULATOR_ALLOCATOR.free(s, x);
 }
 
 static inline int kmem_ptr_validate(struct slab_cache *s, void *x)

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>