Slab initialisation problems on MN10300

Hi,

I'm running into a BUG_ON() when trying to boot an MN10300 arch kernel. The kernel is UP, non-NUMA and is using SLAB. With gdb attached to the kernel, I see the following backtrace:

	(gdb) bt
	#0  0x90258041 in setup_cpu_cache (cachep=0x93c00130) at mm/slab.c:2103
	#1  0x900977d7 in kmem_cache_create (name=0x9026de9d "size-64", size=64, align=16, flags=270336, 
	    ctor=0) at mm/slab.c:2384
	#2  0x9029e959 in kmem_cache_init () at mm/slab.c:1548
	#3  0x902987aa in start_kernel () at init/main.c:618
	#4  0x9000122f in __no_parameters () at arch/mn10300/kernel/head.S:209
	#5  0x9000122f in __no_parameters () at arch/mn10300/kernel/head.S:209

The offending line is this:

		} else {
			int node;
			for_each_online_node(node) {
				cachep->nodelists[node] =
				    kmalloc_node(sizeof(struct kmem_list3),
						GFP_KERNEL, node);

				kmem_list3_init(cachep->nodelists[node]);
			}
		}

This is line 2103 of mm/slab.c in setup_cpu_cache().

Using gdb, I can see that node is 0 and cachep->nodelists[0] is NULL before the kmalloc_node().

Looking in malloc_sizes[], I see:

	(gdb) p malloc_sizes[0]
	$18 = {cs_size = 0x20, cs_cachep = 0x93c000e0}
	(gdb) p malloc_sizes[1]
	$19 = {cs_size = 0x40, cs_cachep = 0x0}
	(gdb) p malloc_sizes[2]
	$20 = {cs_size = 0x60, cs_cachep = 0x0}
	(gdb) p malloc_sizes[3]
	$21 = {cs_size = 0x80, cs_cachep = 0x0}

and sizeof(struct kmem_list3) is 52, which is going to get rounded up to 64 by kmalloc_node(). This means that it's going to attempt to allocate out of the 64-byte kmalloc slab, which is what the kernel is currently setting up, so the allocation fails.

I have to wonder if this comment in setup_cpu_cache() is actually correct:

/*
* Note: the first kmem_cache_create must create the cache
* that's used by kmalloc(24), otherwise the creation of
* further caches will BUG().
*/

Perhaps it's no longer 24, but something bigger. The first pass through setup_cpu_cache() is done for the 32-byte kmalloc slab, with g_cpucache_up set to NONE. The second pass is done for the 64-byte kmalloc slab with g_cpucache_up set to PARTIAL_L3. It is the second pass that fails.

The first pass is special cased. malloc_sizes[0]->cs_cachep is set after the first pass.

The second pass calls kmalloc() on sizeof(struct arraycache_init), which is 20 and succeeds. It then calls kmalloc_node() on sizeof(struct kmem_list3), which is 52 and fails.

David

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Hi David,

What kernel version is this?

size=64, align=16, flags=270336,

So we've already set up caches for struct arraycache_init (INDEX_AC) and struct kmem_list3 (INDEX_L3) here and trying to initialize rest of the caches.

But then:

doesn't make any sense as we should have already initialized the cache for sizeof(struct kmem_list3) (denoted by INDEX_L3).

Also:

If you didn't see PARTIAL_AC state at all, SLAB thinks INDEX_AC and INDEX_L3 are equal. However,

would put struct arraycache_init to kmalloc-32 and struct kmem_list3 to kmalloc-64. So are INDEX_AC and INDEX_L3 really equivalent? To which cache do they refer to?

And if this broke recently, you might want to try and see if commit 556a169dab38b5100df6f4a45b655dddd3db94c1 ("slab: fix bootstrap on memoryless node") is at fault here by reverting it.

Pekka

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Hi,

Hmm, I double-checked the patch and it probably isn't the cause here. It's just that we haven't changed SLAB bootstrap all that much except for this patch. One thing that I thought of was ARCH_KMALLOC_MINALIGN which is set to some fairly big values on some MIPS architectures (MN10300 is one, right?) but reading the code I wasn't able to see what could go wrong with that either...

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

No. MN10300 is not MIPS.

David

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

(gdb) p sizeof(struct arraycache_init)
$1 = 20
(gdb) p sizeof(struct kmem_list3)
$2 = 52

However, the compiler has eliminated the test:

		if (INDEX_AC == INDEX_L3)

even though it's compiled with -O0.

This is odd. I'll have to investigate the preprocessor output.

Well, the MN10300 arch worked in -mm, but no longer works now that the patches have been merged into Linus's tree. Bisecting is probably not an option.

Thanks, anyway. I've got something to investigate.

David

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

What's PAGE_SIZE for the architecture? If it's something other than 4KB, the size 32 cache is not there which makes both use the 64 one. However, that should work too, so maybe there's some GCC bug here for your architecture?

Pekka

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

It's 4K.

David

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Ah... The problem is that index_of() behaves differently under -O0 rather than -O1, -O2 or -Os.

I was using -O0 so that I could debug another problem using GDB on the kernel. However, this appears to mean that __builtin_constant_p() inside an inline function is always false, even if the function is actually inlined because of __always_inline.

I'd commented out the __bad_size() calls because they went to places that don't exist, and so the -O0 kernel wouldn't link.

David

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>