Skip to contentSkip to conversation

ARCH_SLAB_MINALIGN for 2.6.10-rc3

PATCH Manfred Spraul 2 participants 7 messages Dec 5, 2004 – Dec 13, 2004 Memory management

lore
Share & tools
lore
#0 lore raw

Hi Paul,

Quoted 4 lines
include/asm-sh64/uaccess.h +0 -0
@@ -313,6 +313,12 @@
   sh64 at the moment). */

include/asm-sh64/uaccess.h line 323

Quoted 10 lines
include/asm-sh64/uaccess.h +6 -0
 #define ARCH_KMALLOC_MINALIGN 8
 
+/*
+ * We want 8-byte alignment for the slab caches as well, otherwise we have
+ * the same BYTES_PER_WORD (sizeof(void *)) min align in kmem_cache_create().
+ */
+#define ARCH_SLAB_MINALIGN 8
+
 
 

Could you make that dependant on !CONFIG_DEBUG_SLAB? Setting align to a non-zero value disables some debug code.

The rest is fine with me.

--
Manfred

#1 lore raw

Hi Manfred,

Quoted 6 lines · Manfred Spraul
Quoted 6 lines
include/asm-sh64/uaccess.h +0 -0
@@ -313,6 +313,12 @@
  sh64 at the moment). */
 #define ARCH_KMALLOC_MINALIGN 8
 
Quoted 12 lines

include/asm-sh64/uaccess.h

Quoted 9 lines
include/asm-sh64/uaccess.h +3 -0
+/*
+ * We want 8-byte alignment for the slab caches as well, otherwise we have
+ * the same BYTES_PER_WORD (sizeof(void *)) min align in 

kmem_cache_create().
+ */

include/asm-sh64/uaccess.h +2 -0
+#define ARCH_SLAB_MINALIGN 8
+

Could you make that dependant on !CONFIG_DEBUG_SLAB? Setting align to a non-zero value disables some debug code.

align is only being set to ARCH_SLAB_MINALIGN in kmem_cache_create() where it is otherwise being set to BYTES_PER_WORD as a default. Unless I am missing something, that will always set it non-zero irregardless of whether ARCH_SLAB_MINALIGN is set.

Are you suggesting that ARCH_SLAB_MINALIGN be set to 0 in the CONFIG_DEBUG_SLAB case? In that case, the check should be in mm/slab.c and not in the arch-specific code (as any other platform wishing to have fixed slab min alignment would have to do the same checks).

#2 lore raw
Quoted 26 lines · Paul Mundt

Hi Manfred,

Quoted 6 lines · Manfred Spraul
Quoted 6 lines
include/asm-sh64/uaccess.h +0 -0
@@ -313,6 +313,12 @@
 sh64 at the moment). */
 #define ARCH_KMALLOC_MINALIGN 8
 
Quoted 15 lines

include/asm-sh64/uaccess.h

Quoted 11 lines
include/asm-sh64/uaccess.h +3 -0
+/*
+ * We want 8-byte alignment for the slab caches as well, otherwise we have
+ * the same BYTES_PER_WORD (sizeof(void *)) min align in 

kmem_cache_create().
+ */

include/asm-sh64/uaccess.h +2 -0
+#define ARCH_SLAB_MINALIGN 8
+

Could you make that dependant on !CONFIG_DEBUG_SLAB? Setting align to a non-zero value disables some debug code.

Quoted 7 lines

align is only being set to ARCH_SLAB_MINALIGN in kmem_cache_create() where it is otherwise being set to BYTES_PER_WORD as a default. Unless I am missing something, that will always set it non-zero irregardless of whether ARCH_SLAB_MINALIGN is set.

No, you are right. I didn't read the source carefully enough.
Now that I have reread it, I see one problem:
ARCH_KMALLOC_MINALIGN is a hard limit: It's always honored, the only exception is that values larger than the kmalloc block size are ignored. I.e. _MINALIGN 32 guarantees that the objects are 32-byte aligned (since the smallest block size is 32-bytes). The define was added, because some archs really need a certain alignment, otherwise they won't boot. The normal alignment for kmalloc caches is cache line alignment, except with CONFIG_DEBUG_SLAB, then it's word alignment.

With your patch, ARCH_SLAB_MINALIGN is not a hard limit: A few lines further down align is reset to word size if SLAB_RED_ZONE is set. I don't like the asymmetry - it just asks for trouble.

I must think about it. Perhaps just rename ARCH_SLAB_MINALIGN to ARCH_SLAB_DEFAULTALIGN.

--
Manfred

#3 lore raw

Hi Manfred,

Quoted 4 lines · Manfred Spraul

With your patch, ARCH_SLAB_MINALIGN is not a hard limit: A few lines further down align is reset to word size if SLAB_RED_ZONE is set. I don't like the asymmetry - it just asks for trouble.

Yes, that's true. I don't see much of a point in leaving it as BYTES_PER_WORD in the SLAB_RED_ZONE case, at least this wasn't intentional. Would you accept ARCH_SLAB_MINALIGN if this was set regardless of whether SLAB_RED_ZONE is set or not?

I suppose we can live with align being 0 in the CONFIG_DEBUG_SLAB case as the unaligned accesses are not fatal..

#4 lore raw

Hi Paul,

Sorry for the late reply, attached is my proposal:
I've added the ARCH_SLAB_MINALIGN flag, together with some documentation and a small restructuring. What do you think? It's just the mm/slab.c change, you would have to add

#ifndef CONFIG_DEBUG_SLAB
#define ARCH_SLAB_MINALIGN 8
#endif

into your sh64 header files. ARCH_SLAB_MINALIGN includes ARCH_KMALLOC_MINALIGN, so you do not have to set that flag. It doesn't hurt, though.

Not really tested - it boots on x86, but that probably doesn't count.

--
Manfred

mm/slab.c +55 -25
@@ -128,9 +128,28 @@
 #endif
 
 #ifndef ARCH_KMALLOC_MINALIGN
+/*
+ * Enforce a minimum alignment for the kmalloc caches.
+ * Usually, the kmalloc caches are cache_line_size() aligned, except when
+ * DEBUG and FORCED_DEBUG are enabled, then they are BYTES_PER_WORD aligned.
+ * Some archs want to perform DMA into kmalloc caches and need a guaranteed
+ * alignment larger than BYTES_PER_WORD. ARCH_KMALLOC_MINALIGN allows that.
+ * Note that this flag disables some debug features.
+ */
 #define ARCH_KMALLOC_MINALIGN 0
 #endif
 
+#ifndef ARCH_SLAB_MINALIGN
+/*
+ * Enforce a minimum alignment for all caches.
+ * Intended for archs that get misalignment faults even for BYTES_PER_WORD
+ * aligned buffers.
+ * If possible: Do not enable this flag for CONFIG_DEBUG_SLAB, it disables
+ * some debug features.
+ */
+#define ARCH_SLAB_MINALIGN 0
+#endif
+
 #ifndef ARCH_KMALLOC_FLAGS
 #define ARCH_KMALLOC_FLAGS SLAB_HWCACHE_ALIGN
 #endif
@@ -1172,7 +1191,7 @@
 	unsigned long flags, void (*ctor)(void*, kmem_cache_t *, unsigned long),
 	void (*dtor)(void*, kmem_cache_t *, unsigned long))
 {
-	size_t left_over, slab_size;
+	size_t left_over, slab_size, ralign;
 	kmem_cache_t *cachep = NULL;
 
 	/*
@@ -1222,24 +1241,44 @@
 	if (flags & ~CREATE_MASK)
 		BUG();
 
-	if (align) {
-		/* combinations of forced alignment and advanced debugging is
-		 * not yet implemented.
+	/* Check that size is in terms of words.  This is needed to avoid
+	 * unaligned accesses for some archs when redzoning is used, and makes
+	 * sure any on-slab bufctl's are also correctly aligned.
+	 */
+	if (size & (BYTES_PER_WORD-1)) {
+		size += (BYTES_PER_WORD-1);
+		size &= ~(BYTES_PER_WORD-1);
+	}
+	
+	/* calculate out the final buffer alignment: */
+	/* 1) arch recommendation: can be overridden for debug */
+	if (flags & SLAB_HWCACHE_ALIGN) {
+		/* Default alignment: as specified by the arch code.
+		 * Except if an object is really small, then squeeze multiple
+		 * objects into one cacheline.
 		 */
-		flags &= ~(SLAB_RED_ZONE|SLAB_STORE_USER);
+		ralign = cache_line_size();
+		while (size <= ralign/2)
+			ralign /= 2;
 	} else {
-		if (flags & SLAB_HWCACHE_ALIGN) {
-			/* Default alignment: as specified by the arch code.
-			 * Except if an object is really small, then squeeze multiple
-			 * into one cacheline.
-			 */
-			align = cache_line_size();
-			while (size <= align/2)
-				align /= 2;
-		} else {
-			align = BYTES_PER_WORD;
-		}
+		ralign = BYTES_PER_WORD;
+	}
+	/* 2) arch mandated alignment: disables debug if necessary */
+	if (ralign < ARCH_SLAB_MINALIGN) {
+		ralign = ARCH_SLAB_MINALIGN;
+		if (ralign > BYTES_PER_WORD)
+			flags &= ~(SLAB_RED_ZONE|SLAB_STORE_USER);
+	}
+	/* 3) caller mandated alignment: disables debug if necessary */
+	if (ralign < align) {
+		ralign = align;
+		if (ralign > BYTES_PER_WORD)
+			flags &= ~(SLAB_RED_ZONE|SLAB_STORE_USER);
 	}
+	/* 4) Store it. Note that the debug code below can reduce
+	 *    the alignment to BYTES_PER_WORD.
+	 */
+	align = ralign;
 
 	/* Get cache's description obj. */
 	cachep = (kmem_cache_t *) kmem_cache_alloc(&cache_cache, SLAB_KERNEL);
@@ -1247,15 +1286,6 @@
 		goto opps;
 	memset(cachep, 0, sizeof(kmem_cache_t));
 
-	/* Check that size is in terms of words.  This is needed to avoid
-	 * unaligned accesses for some archs when redzoning is used, and makes
-	 * sure any on-slab bufctl's are also correctly aligned.
-	 */
-	if (size & (BYTES_PER_WORD-1)) {
-		size += (BYTES_PER_WORD-1);
-		size &= ~(BYTES_PER_WORD-1);
-	}
-	
 #if DEBUG
 	cachep->reallen = size;
 
#5 lore raw

Hi Manfred,

Quoted 5 lines · Manfred Spraul

Sorry for the late reply, attached is my proposal:
I've added the ARCH_SLAB_MINALIGN flag, together with some documentation and a small restructuring.
What do you think?

Looks fine to me, just tested on sh64 and it works ok.

Quoted 4 lines

#ifndef CONFIG_DEBUG_SLAB
#define ARCH_SLAB_MINALIGN 8
#endif

Right now ARCH_KMALLOC_MINALIGN is set unconditionally on sh64, so it seems that we lose some debug features there. However, even with ARCH_SLAB_MINALIGN being set to non-zero, redzoning and slab poisoning still seem to be functional to some extent.

For instance, I've been using the following patch and this did help pin down a rather irritating bug in the sh64 switch_to().

Is there any reason not to wrap redzoning to ARCH_SLAB_MINALIGN (and set it to BYTES_PER_WORD by default). It seems at least that the only thing BYTES_PER_WORD is needed for in the redzoning case is to determine where to place the second marker. I can see why this would be problematic with dynamic slab alignment, but when it is fixed at compile time there shouldn't be anything prohibiting the use of a non-BYTES_PER_WORD value.

We can live with the unaligned accesses in the CONFIG_DEBUG_SLAB case, but it would still be nice to at least have some partial redzoning and poisoning with a forced alignment.

mm/slab.c +26 -20
@@ -135,6 +135,10 @@
 #define ARCH_KMALLOC_FLAGS SLAB_HWCACHE_ALIGN
 #endif
 
+#ifndef ARCH_SLAB_MINALIGN
+#define ARCH_SLAB_MINALIGN	BYTES_PER_WORD
+#endif
+
 /* Legal flag mask for kmem_cache_create(). */
 #if DEBUG
 # define CREATE_MASK	(SLAB_DEBUG_INITIAL | SLAB_RED_ZONE | \
@@ -404,14 +408,16 @@
 
 /* memory layout of objects:
  * 0		: objp
- * 0 .. cachep->dbghead - BYTES_PER_WORD - 1: padding. This ensures that
- * 		the end of an object is aligned with the end of the real
- * 		allocation. Catches writes behind the end of the allocation.
- * cachep->dbghead - BYTES_PER_WORD .. cachep->dbghead - 1:
- * 		redzone word.
+ * 0 .. cachep->dbghead - ARCH_SLAB_MINALIGN - 1: padding. This ensures that
+ *		the end of an object is aligned with the end of the real
+ *		allocation. Catches writes behind the end of the allocation.
+ * cachep->dbghead - ARCH_SLAB_MINALIGN .. cachep->dbghead - 1:
+ *		redzone word.
  * cachep->dbghead: The real object.
- * cachep->objsize - 2* BYTES_PER_WORD: redzone word [BYTES_PER_WORD long]
- * cachep->objsize - 1* BYTES_PER_WORD: last caller address [BYTES_PER_WORD long]
+ * cachep->objsize - 2* ARCH_SLAB_MINALIGN:
+ *		redzone word [ARCH_SLAB_MINALIGN long]
+ * cachep->objsize - 1* ARCH_SLAB_MINALIGN:
+ *		last caller address [ARCH_SLAB_MINALIGN long]
  */
 static int obj_dbghead(kmem_cache_t *cachep)
 {
@@ -426,21 +432,21 @@
 static unsigned long *dbg_redzone1(kmem_cache_t *cachep, void *objp)
 {
 	BUG_ON(!(cachep->flags & SLAB_RED_ZONE));
-	return (unsigned long*) (objp+obj_dbghead(cachep)-BYTES_PER_WORD);
+	return (unsigned long*) (objp+obj_dbghead(cachep)-ARCH_SLAB_MINALIGN);
 }
 
 static unsigned long *dbg_redzone2(kmem_cache_t *cachep, void *objp)
 {
 	BUG_ON(!(cachep->flags & SLAB_RED_ZONE));
 	if (cachep->flags & SLAB_STORE_USER)
-		return (unsigned long*) (objp+cachep->objsize-2*BYTES_PER_WORD);
-	return (unsigned long*) (objp+cachep->objsize-BYTES_PER_WORD);
+		return (unsigned long*) (objp+cachep->objsize-2*ARCH_SLAB_MINALIGN);
+	return (unsigned long*) (objp+cachep->objsize-ARCH_SLAB_MINALIGN);
 }
 
 static void **dbg_userword(kmem_cache_t *cachep, void *objp)
 {
 	BUG_ON(!(cachep->flags & SLAB_STORE_USER));
-	return (void**)(objp+cachep->objsize-BYTES_PER_WORD);
+	return (void**)(objp+cachep->objsize-ARCH_SLAB_MINALIGN);
 }
 
 #else
@@ -1204,7 +1210,7 @@
 	 * above the next power of two: caches with object sizes just above a
 	 * power of two have a significant amount of internal fragmentation.
 	 */
-	if ((size < 4096 || fls(size-1) == fls(size-1+3*BYTES_PER_WORD)))
+	if ((size < 4096 || fls(size-1) == fls(size-1+3*ARCH_SLAB_MINALIGN)))
 		flags |= SLAB_RED_ZONE|SLAB_STORE_USER;
 	if (!(flags & SLAB_DESTROY_BY_RCU))
 		flags |= SLAB_POISON;
@@ -1237,7 +1243,7 @@
 			while (size <= align/2)
 				align /= 2;
 		} else {
-			align = BYTES_PER_WORD;
+			align = ARCH_SLAB_MINALIGN;
 		}
 	}
 
@@ -1255,25 +1261,25 @@
 		size += (BYTES_PER_WORD-1);
 		size &= ~(BYTES_PER_WORD-1);
 	}
-	
+
 #if DEBUG
 	cachep->reallen = size;
 
 	if (flags & SLAB_RED_ZONE) {
 		/* redzoning only works with word aligned caches */
-		align = BYTES_PER_WORD;
+		align = ARCH_SLAB_MINALIGN;
 
 		/* add space for red zone words */
-		cachep->dbghead += BYTES_PER_WORD;
-		size += 2*BYTES_PER_WORD;
+		cachep->dbghead += ARCH_SLAB_MINALIGN;
+		size += 2*ARCH_SLAB_MINALIGN;
 	}
 	if (flags & SLAB_STORE_USER) {
 		/* user store requires word alignment and
 		 * one word storage behind the end of the real
 		 * object.
 		 */
-		align = BYTES_PER_WORD;
-		size += BYTES_PER_WORD;
+		align = ARCH_SLAB_MINALIGN;
+		size += ARCH_SLAB_MINALIGN;
 	}
 #if FORCED_DEBUG && defined(CONFIG_DEBUG_PAGEALLOC)
 	if (size > 128 && cachep->reallen > cache_line_size() && size < PAGE_SIZE) {
@@ -2292,7 +2298,7 @@
 {
 	unsigned long addr = (unsigned long) ptr;
 	unsigned long min_addr = PAGE_OFFSET;
-	unsigned long align_mask = BYTES_PER_WORD-1;
+	unsigned long align_mask = ARCH_SLAB_MINALIGN-1;
 	unsigned long size = cachep->objsize;
 	struct page *page;
 
#6 lore raw
Quoted 5 lines · Paul Mundt

For instance, I've been using the following patch and this did help pin down a rather irritating bug in the sh64 switch_to().

[snip - useful patch]

I agree with your patch. But I think the change is independant, so it should remain a seperate patch. I've just sent my patch to Andrew for merging, could you send your patch to Andrew after my change was merged?

--
Manfred