ARCH_SLAB_MINALIGN for 2.6.10-rc3
Hi Paul,
Quoted 4 lines
include/asm-sh64/uaccess.h +0
-0@@ -313,6 +313,12 @@sh64 at the moment). */
include/asm-sh64/uaccess.h line 323
Quoted 10 lines
include/asm-sh64/uaccess.h +6
-0#define ARCH_KMALLOC_MINALIGN 8 +/* +* We want 8-byte alignment for the slab caches as well, otherwise we have +* the same BYTES_PER_WORD (sizeof(void *)) min align in kmem_cache_create(). +*/ +#define ARCH_SLAB_MINALIGN 8 +
Could you make that dependant on !CONFIG_DEBUG_SLAB? Setting align to a non-zero value disables some debug code.
The rest is fine with me.
--
Manfred
Hi Manfred,
Quoted 6 lines · Manfred Spraul
Quoted 6 lines
include/asm-sh64/uaccess.h +0
-0@@ -313,6 +313,12 @@sh64 at the moment). */ #define ARCH_KMALLOC_MINALIGN 8
Quoted 12 lines
include/asm-sh64/uaccess.h
Quoted 9 lines
include/asm-sh64/uaccess.h +3
-0+/* +* We want 8-byte alignment for the slab caches as well, otherwise we have +* the same BYTES_PER_WORD (sizeof(void *)) min align in
kmem_cache_create().
+ */include/asm-sh64/uaccess.h +2
-0+#define ARCH_SLAB_MINALIGN 8 +Could you make that dependant on !CONFIG_DEBUG_SLAB? Setting align to a non-zero value disables some debug code.
align is only being set to ARCH_SLAB_MINALIGN in kmem_cache_create() where it is otherwise being set to BYTES_PER_WORD as a default. Unless I am missing something, that will always set it non-zero irregardless of whether ARCH_SLAB_MINALIGN is set.
Are you suggesting that ARCH_SLAB_MINALIGN be set to 0 in the CONFIG_DEBUG_SLAB case? In that case, the check should be in mm/slab.c and not in the arch-specific code (as any other platform wishing to have fixed slab min alignment would have to do the same checks).
Quoted 26 lines · Paul Mundt
Hi Manfred,
Quoted 6 lines · Manfred Spraul
Quoted 6 lines
include/asm-sh64/uaccess.h +0
-0@@ -313,6 +313,12 @@sh64 at the moment). */ #define ARCH_KMALLOC_MINALIGN 8Quoted 15 lines
include/asm-sh64/uaccess.h
Quoted 11 lines
include/asm-sh64/uaccess.h +3
-0+/* +* We want 8-byte alignment for the slab caches as well, otherwise we have +* the same BYTES_PER_WORD (sizeof(void *)) min align in
kmem_cache_create().
+ */include/asm-sh64/uaccess.h +2
-0+#define ARCH_SLAB_MINALIGN 8 +Could you make that dependant on !CONFIG_DEBUG_SLAB? Setting align to a non-zero value disables some debug code.
Quoted 7 lines
align is only being set to
ARCH_SLAB_MINALIGNinkmem_cache_create()where it is otherwise being set toBYTES_PER_WORDas a default. Unless I am missing something, that will always set it non-zero irregardless of whetherARCH_SLAB_MINALIGNis set.
No, you are right. I didn't read the source carefully enough.
Now that I have reread it, I see one problem:ARCH_KMALLOC_MINALIGN is a hard limit: It's always honored, the only exception is that values larger than the kmalloc block size are ignored. I.e. _MINALIGN 32 guarantees that the objects are 32-byte aligned (since the smallest block size is 32-bytes). The define was added, because some archs really need a certain alignment, otherwise they won't boot. The normal alignment for kmalloc caches is cache line alignment, except with CONFIG_DEBUG_SLAB, then it's word alignment.
With your patch, ARCH_SLAB_MINALIGN is not a hard limit: A few lines further down align is reset to word size if SLAB_RED_ZONE is set. I don't like the asymmetry - it just asks for trouble.
I must think about it. Perhaps just rename ARCH_SLAB_MINALIGN to ARCH_SLAB_DEFAULTALIGN.
--
Manfred
Hi Manfred,
Quoted 4 lines · Manfred Spraul
With your patch,
ARCH_SLAB_MINALIGNis not a hard limit: A few lines further down align is reset to word size ifSLAB_RED_ZONEis set. I don't like the asymmetry - it just asks for trouble.
Yes, that's true. I don't see much of a point in leaving it as BYTES_PER_WORD in the SLAB_RED_ZONE case, at least this wasn't intentional. Would you accept ARCH_SLAB_MINALIGN if this was set regardless of whether SLAB_RED_ZONE is set or not?
I suppose we can live with align being 0 in the CONFIG_DEBUG_SLAB case as the unaligned accesses are not fatal..
Hi Paul,
Sorry for the late reply, attached is my proposal:
I've added the ARCH_SLAB_MINALIGN flag, together with some documentation and a small restructuring. What do you think? It's just the mm/slab.c change, you would have to add
#ifndef CONFIG_DEBUG_SLAB
#define ARCH_SLAB_MINALIGN 8
#endif
into your sh64 header files. ARCH_SLAB_MINALIGN includes ARCH_KMALLOC_MINALIGN, so you do not have to set that flag. It doesn't hurt, though.
Not really tested - it boots on x86, but that probably doesn't count.
--
Manfred
mm/slab.c +55 -25
@@ -128,9 +128,28 @@#endif #ifndef ARCH_KMALLOC_MINALIGN +/* +* Enforce a minimum alignment for the kmalloc caches. +* Usually, the kmalloc caches are cache_line_size() aligned, except when +* DEBUG and FORCED_DEBUG are enabled, then they are BYTES_PER_WORD aligned. +* Some archs want to perform DMA into kmalloc caches and need a guaranteed +* alignment larger than BYTES_PER_WORD. ARCH_KMALLOC_MINALIGN allows that. +* Note that this flag disables some debug features. +*/ #define ARCH_KMALLOC_MINALIGN 0#endif +#ifndef ARCH_SLAB_MINALIGN +/* +* Enforce a minimum alignment for all caches. +* Intended for archs that get misalignment faults even for BYTES_PER_WORD +* aligned buffers. +* If possible: Do not enable this flag for CONFIG_DEBUG_SLAB, it disables +* some debug features. +*/ +#define ARCH_SLAB_MINALIGN 0 +#endif +#ifndef ARCH_KMALLOC_FLAGS #define ARCH_KMALLOC_FLAGS SLAB_HWCACHE_ALIGN#endif
@@ -1172,7 +1191,7 @@ unsigned long flags, void (*ctor)(void*, kmem_cache_t *, unsigned long), void (*dtor)(void*, kmem_cache_t *, unsigned long)) { - size_t left_over, slab_size; + size_t left_over, slab_size, ralign;kmem_cache_t *cachep =NULL ;/*
@@ -1222,24 +1241,44 @@if (flags & ~CREATE_MASK )BUG (); -if (align) { -/* combinations of forced alignment and advanced debugging is -* not yet implemented. +/* Check that size is in terms of words. This is needed to avoid +* unaligned accesses for some archs when redzoning is used, and makes +* sure any on-slab bufctl's are also correctly aligned. +*/ +if (size & (BYTES_PER_WORD -1 )) { + size += (BYTES_PER_WORD -1 ); + size &= ~(BYTES_PER_WORD -1 ); + } + +/* calculate out the final buffer alignment: */ +/* 1) arch recommendation: can be overridden for debug */ +if (flags &SLAB_HWCACHE_ALIGN ) { +/* Default alignment: as specified by the arch code. +* Except if an object is really small, then squeeze multiple +* objects into one cacheline. */ - flags &= ~(SLAB_RED_ZONE |SLAB_STORE_USER ); + ralign =cache_line_size (); +while (size <= ralign/2 ) + ralign /=2 ; }else { -if (flags &SLAB_HWCACHE_ALIGN ) { -/* Default alignment: as specified by the arch code. -* Except if an object is really small, then squeeze multiple -* into one cacheline. -*/ - align =cache_line_size (); -while (size <= align/2 ) - align /=2 ; - }else { - align =BYTES_PER_WORD ; - } + ralign =BYTES_PER_WORD ; + } +/* 2) arch mandated alignment: disables debug if necessary */ +if (ralign <ARCH_SLAB_MINALIGN ) { + ralign =ARCH_SLAB_MINALIGN ; +if (ralign >BYTES_PER_WORD ) + flags &= ~(SLAB_RED_ZONE |SLAB_STORE_USER ); + } +/* 3) caller mandated alignment: disables debug if necessary */ +if (ralign < align) { + ralign = align; +if (ralign >BYTES_PER_WORD ) + flags &= ~(SLAB_RED_ZONE |SLAB_STORE_USER ); } +/* 4) Store it. Note that the debug code below can reduce +* the alignment to BYTES_PER_WORD. +*/ + align = ralign;/* Get cache's description obj. */ cachep = (kmem_cache_t *)kmem_cache_alloc (&cache_cache,SLAB_KERNEL );
@@ -1247,15 +1286,6 @@ goto opps;memset (cachep,0 ,sizeof (kmem_cache_t)); -/* Check that size is in terms of words. This is needed to avoid -* unaligned accesses for some archs when redzoning is used, and makes -* sure any on-slab bufctl's are also correctly aligned. -*/ -if (size & (BYTES_PER_WORD -1 )) { - size += (BYTES_PER_WORD -1 ); - size &= ~(BYTES_PER_WORD -1 ); - } -#if DEBUG cachep->reallen = size;
Hi Manfred,
Quoted 5 lines · Manfred Spraul
Sorry for the late reply, attached is my proposal:
I've added theARCH_SLAB_MINALIGNflag, together with some documentation and a small restructuring.
What do you think?
Looks fine to me, just tested on sh64 and it works ok.
Quoted 4 lines
#ifndef
CONFIG_DEBUG_SLAB
#defineARCH_SLAB_MINALIGN8
#endif
Right now ARCH_KMALLOC_MINALIGN is set unconditionally on sh64, so it seems that we lose some debug features there. However, even with ARCH_SLAB_MINALIGN being set to non-zero, redzoning and slab poisoning still seem to be functional to some extent.
For instance, I've been using the following patch and this did help pin down a rather irritating bug in the sh64 switch_to().
Is there any reason not to wrap redzoning to ARCH_SLAB_MINALIGN (and set it to BYTES_PER_WORD by default). It seems at least that the only thing BYTES_PER_WORD is needed for in the redzoning case is to determine where to place the second marker. I can see why this would be problematic with dynamic slab alignment, but when it is fixed at compile time there shouldn't be anything prohibiting the use of a non-BYTES_PER_WORD value.
We can live with the unaligned accesses in the CONFIG_DEBUG_SLAB case, but it would still be nice to at least have some partial redzoning and poisoning with a forced alignment.
mm/slab.c +26 -20
@@ -135,6 +135,10 @@#define ARCH_KMALLOC_FLAGS SLAB_HWCACHE_ALIGN#endif +#ifndef ARCH_SLAB_MINALIGN +#define ARCH_SLAB_MINALIGN BYTES_PER_WORD +#endif +/* Legal flag mask for kmem_cache_create(). */ #if DEBUG # define CREATE_MASK (SLAB_DEBUG_INITIAL | SLAB_RED_ZONE | \
@@ -404,14 +408,16 @@/* memory layout of objects: * 0 : objp -* 0 .. cachep->dbghead - BYTES_PER_WORD - 1: padding. This ensures that -* the end of an object is aligned with the end of the real -* allocation. Catches writes behind the end of the allocation. -* cachep->dbghead - BYTES_PER_WORD .. cachep->dbghead - 1: -* redzone word. +* 0 .. cachep->dbghead - ARCH_SLAB_MINALIGN - 1: padding. This ensures that +* the end of an object is aligned with the end of the real +* allocation. Catches writes behind the end of the allocation. +* cachep->dbghead - ARCH_SLAB_MINALIGN .. cachep->dbghead - 1: +* redzone word. * cachep->dbghead: The real object. -* cachep->objsize - 2* BYTES_PER_WORD: redzone word [BYTES_PER_WORD long] -* cachep->objsize - 1* BYTES_PER_WORD: last caller address [BYTES_PER_WORD long] +* cachep->objsize - 2* ARCH_SLAB_MINALIGN: +* redzone word [ARCH_SLAB_MINALIGN long] +* cachep->objsize - 1* ARCH_SLAB_MINALIGN: +* last caller address [ARCH_SLAB_MINALIGN long] */ static int obj_dbghead (kmem_cache_t *cachep) {
@@ -426,21 +432,21 @@static unsigned long *dbg_redzone1 (kmem_cache_t *cachep,void *objp) {BUG_ON (!(cachep->flags &SLAB_RED_ZONE )); -return (unsigned long *) (objp+obj_dbghead (cachep)-BYTES_PER_WORD ); +return (unsigned long *) (objp+obj_dbghead (cachep)-ARCH_SLAB_MINALIGN ); }static unsigned long *dbg_redzone2 (kmem_cache_t *cachep,void *objp) {BUG_ON (!(cachep->flags &SLAB_RED_ZONE ));if (cachep->flags &SLAB_STORE_USER ) -return (unsigned long *) (objp+cachep->objsize-2 *BYTES_PER_WORD ); -return (unsigned long *) (objp+cachep->objsize-BYTES_PER_WORD ); +return (unsigned long *) (objp+cachep->objsize-2 *ARCH_SLAB_MINALIGN ); +return (unsigned long *) (objp+cachep->objsize-ARCH_SLAB_MINALIGN ); }static void **dbg_userword (kmem_cache_t *cachep,void *objp) {BUG_ON (!(cachep->flags &SLAB_STORE_USER )); -return (void **)(objp+cachep->objsize-BYTES_PER_WORD ); +return (void **)(objp+cachep->objsize-ARCH_SLAB_MINALIGN ); }#else
@@ -1204,7 +1210,7 @@* above the next power of two: caches with object sizes just above a * power of two have a significant amount of internal fragmentation. */ -if ((size <4096 ||fls (size-1 ) ==fls (size-1 +3 *BYTES_PER_WORD ))) +if ((size <4096 ||fls (size-1 ) ==fls (size-1 +3 *ARCH_SLAB_MINALIGN ))) flags |=SLAB_RED_ZONE |SLAB_STORE_USER ;if (!(flags &SLAB_DESTROY_BY_RCU )) flags |=SLAB_POISON ;
@@ -1237,7 +1243,7 @@while (size <= align/2 ) align /=2 ; }else { - align =BYTES_PER_WORD ; + align =ARCH_SLAB_MINALIGN ; } }
@@ -1255,25 +1261,25 @@ size += (BYTES_PER_WORD -1 ); size &= ~(BYTES_PER_WORD -1 ); } - + #if DEBUG cachep->reallen = size; if (flags & SLAB_RED_ZONE) { /* redzoning only works with word aligned caches */ - align = BYTES_PER_WORD; + align = ARCH_SLAB_MINALIGN; /* add space for red zone words */ - cachep->dbghead += BYTES_PER_WORD; - size += 2*BYTES_PER_WORD; + cachep->dbghead += ARCH_SLAB_MINALIGN; + size += 2*ARCH_SLAB_MINALIGN; } if (flags & SLAB_STORE_USER) { /* user store requires word alignment and * one word storage behind the end of the real * object. */ - align = BYTES_PER_WORD; - size += BYTES_PER_WORD; + align = ARCH_SLAB_MINALIGN; + size += ARCH_SLAB_MINALIGN; } #if FORCED_DEBUG && defined(CONFIG_DEBUG_PAGEALLOC) if (size > 128 && cachep->reallen > cache_line_size() && size < PAGE_SIZE) {
@@ -2292,7 +2298,7 @@ {unsigned long addr = (unsigned long ) ptr;unsigned long min_addr =PAGE_OFFSET ; -unsigned long align_mask =BYTES_PER_WORD -1 ; +unsigned long align_mask =ARCH_SLAB_MINALIGN -1 ;unsigned long size = cachep->objsize;struct page *page;
Quoted 5 lines · Paul Mundt
For instance, I've been using the following patch and this did help pin down a rather irritating bug in the sh64
switch_to().
[snip - useful patch]
I agree with your patch. But I think the change is independant, so it should remain a seperate patch. I've just sent my patch to Andrew for merging, could you send your patch to Andrew after my change was merged?
--
Manfred