mm/slab: ppc: ubi: kmalloc_slab WARNING / PPC + UBI driver

Hello guys,

on a PPC 32-Bit board with a Linux Kernel v3.10.0 I see trouble with kmalloc_slab. Basically at system startup, something request a size of 8388608 b, but KMALLOC_MAX_SIZE has 4194304 b in our case. It points a WARNING at:
..
NIP [c0099fec] kmalloc_slab+0x60/0xe8
LR [c0099fd4] kmalloc_slab+0x48/0xe8

Call Trace:
[ccd3be60] [c0099fd4] kmalloc_slab+0x48/0xe8 (unreliable)
[ccd3be70] [c00ae650] __kmalloc+0x20/0x1b4
[ccd3be90] [c00d46f4] seq_read+0x2a4/0x540
[ccd3bee0] [c00fe09c] proc_reg_read+0x5c/0x90
[ccd3bef0] [c00b4e1c] vfs_read+0xa4/0x150
[ccd3bf10] [c00b500c] SyS_read+0x4c/0x84
[ccd3bf40] [c000be80] ret_from_syscall+0x0/0x3c

..

Do you have any idea how I can analyze where these 8388608 b coming from? Kernel log show that UBI driver tries to do something, but I am unsure where it is in detail.

I enabled in the kernel config additionally this parts
- Debug slab memory allocations
- Verbose BUG() reporting (adds 70K)
- Debug VM
- Debug memory initialisation
- Debug page memory allocations

log snip:
...
UBI: attaching mtd0 to ubi0
UBI: scanning is finished
UBI: attached mtd0 (name "fs", size 47 MiB) to ubi0
UBI: PEB size: 131072 bytes (128 KiB), LEB size: 130944 bytes
UBI: min./max. I/O unit sizes: 1/8, sub-page size 1
UBI: VID header offset: 64 (aligned 64), data offset: 128
UBI: good PEBs: 376, bad PEBs: 0, corrupted PEBs: 0
UBI: user volume: 1, internal volumes: 1, max. volumes count: 128
UBI: max/mean erase counter: 8/5, WL threshold: 4096, image sequence number: 657495904
UBI: available PEBs: 0, total reserved PEBs: 376, PEBs reserved for bad PEB handling: 0
UBI: background thread "ubi_bgt0d" started, PID 892
DEBUG: xxx kmalloc_slab, requested 'size' = 8388608, KMALLOC_MAX_SIZE = 4194304

------------[ cut here ]------------
WARNING: at /var/fpwork/wiebe/newfsm/bld/bld-kernelsources-linux/results/linux/mm/slab_common.c:383
Modules linked in: ubi mddg_post(O) mddg_rpram(O) mddg_system_driver(O) mddg_watchdog(O)
CPU: 0 PID: 900 Comm: hexdump Tainted: G           O 3.10.0-0-sampleversion-fcmd #40
task: cf3e7280 ti: ccd3a000 task.ti: ccd3a000
NIP: c0099fec LR: c0099fd4 CTR: c018b0dc
REGS: ccd3bdb0 TRAP: 0700   Tainted: G           O  (3.10.0-0-sampleversion-fcmd)
MSR: 00029000 <CE,EE,ME>  CR: 22000442  XER: 20000000

GPR00: c0099fd4 ccd3be60 cf3e7280 00000000 d100e501 00000005 00000000 c0372ac0
GPR08: 00000004 00000001 00000000 ccd3be20 22000444 100a24dc 00000000 00000000
GPR16: 10076b54 00000000 fffff000 00000000 ccd3bea0 00000000 00000001 00000400
GPR24: 00000000 00000000 4801c000 c00d46f4 ccd3bf18 000000d0 00800000 000000d0 NIP [c0099fec] kmalloc_slab+0x60/0xe8
LR [c0099fd4] kmalloc_slab+0x48/0xe8

Call Trace:
[ccd3be60] [c0099fd4] kmalloc_slab+0x48/0xe8 (unreliable)
[ccd3be70] [c00ae650] __kmalloc+0x20/0x1b4
[ccd3be90] [c00d46f4] seq_read+0x2a4/0x540
[ccd3bee0] [c00fe09c] proc_reg_read+0x5c/0x90
[ccd3bef0] [c00b4e1c] vfs_read+0xa4/0x150
[ccd3bf10] [c00b500c] SyS_read+0x4c/0x84
[ccd3bf40] [c000be80] ret_from_syscall+0x0/0x3c
--- Exception: c01 at 0xfe63934
    LR = 0xfe1a6a8
Instruction dump:
3884dff0 3863e98c 38840094 3cc00040 4cc63182 481e9fc1 73e90200 38600000
40a20090 3d20c038 89291e69 69290001 <0f090000> 2f890000 41be0078 3d20c038
---[ end trace afdc4720a42f3f3c ]---

UBIFS: recovery needed
UBIFS: recovery deferred
UBIFS: mounted UBI device 0, volume 0, name "flash", R/O mode
UBIFS: LEB size: 130944 bytes (127 KiB), min./max. I/O unit sizes: 8 bytes/8 bytes
UBIFS: FS size: 47532672 bytes (45 MiB, 363 LEBs), journal size 2356992 bytes (2 MiB, 18 LEBs)
UBIFS: reserved for root: 0 bytes (0 KiB)
UBIFS: media format: w4/r0 (latest is w4/r0), UUID 7570E817-F0E4-4BF4-8AB5-552B4C55AF30, small LPT model
UBIFS: completing deferred recovery
UBIFS: background thread "ubifs_bgt0_0" started, PID 962
UBIFS: deferred recovery completed
..

Thanks & BR
Wladislav Wiebe

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Aren't you doing a hexdump on a proc file ?

Matthieu

It comes from the kmalloc in seq_read(). And 8M read from the proc filesystem? Wow. Maybe switch the kmalloc to vmalloc()?

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

This patch will suppress the warnings by using the page allocator wrappers of the slab allocators. These are page sized allocs after all.

Subject: seq_file: Use kmalloc_large for page sized allocation

There is no point in using the slab allocation functions for large page order allocation. Use the kmalloc_large() wrappers which will cause calls to the page alocator instead.

This fixes the warning about large allocs but it will still cause high order allocs to occur that could fail because of memory fragmentation. Maybe switch to vmalloc if we really want to allocate multi megabyte buffers for proc fs?

Index: linux/fs/seq_file.c
===================================================================
@@ -96,7 +96,7 @@ static int traverse(struct seq_file *m,
 		return 0;
 	}
 	if (!m->buf) {
-		m->buf = kmalloc(m->size = PAGE_SIZE, GFP_KERNEL);
+		m->buf = kmalloc_large(m->size = PAGE_SIZE, GFP_KERNEL);
 		if (!m->buf)
 			return -ENOMEM;
 	}

@@ -136,7 +136,7 @@ static int traverse(struct seq_file *m,
 Eoverflow:
 	m->op->stop(m, p);
 	kfree(m->buf);
-	m->buf = kmalloc(m->size <<= 1, GFP_KERNEL);
+	m->buf = kmalloc_large(m->size <<= 1, GFP_KERNEL);
 	return !m->buf ? -ENOMEM : -EAGAIN;
 }
 

@@ -191,7 +191,7 @@ ssize_t seq_read(struct file *file, char
 
 	/* grab buffer if we didn't have one */
 	if (!m->buf) {
-		m->buf = kmalloc(m->size = PAGE_SIZE, GFP_KERNEL);
+		m->buf = kmalloc_large(m->size = PAGE_SIZE, GFP_KERNEL);
 		if (!m->buf)
 			goto Enomem;
 	}

@@ -232,7 +232,7 @@ ssize_t seq_read(struct file *file, char
 			goto Fill;
 		m->op->stop(m, p);
 		kfree(m->buf);
-		m->buf = kmalloc(m->size <<= 1, GFP_KERNEL);
+		m->buf = kmalloc_large(m->size <<= 1, GFP_KERNEL);
 		if (!m->buf)
 			goto Enomem;
 		m->count = 0;

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Crap you cannot do PAGE_SIZE allocations with kmalloc_large. Fails when freeing pages. Need to only do the multiple page allocs with kmalloc_large.

Subject: seq_file: Use kmalloc_large for page sized allocation

There is no point in using the slab allocation functions for large page order allocation. Use kmalloc_large().

This fixes the warning about large allocs but it will still cause large contiguous allocs that could fail because of memory fragmentation.

Index: linux/fs/seq_file.c
===================================================================
@@ -136,7 +136,7 @@ static int traverse(struct seq_file *m,
 Eoverflow:
 	m->op->stop(m, p);
 	kfree(m->buf);
-	m->buf = kmalloc(m->size <<= 1, GFP_KERNEL);
+	m->buf = kmalloc_large(m->size <<= 1, GFP_KERNEL);
 	return !m->buf ? -ENOMEM : -EAGAIN;
 }
 

@@ -232,7 +232,7 @@ ssize_t seq_read(struct file *file, char
 			goto Fill;
 		m->op->stop(m, p);
 		kfree(m->buf);
-		m->buf = kmalloc(m->size <<= 1, GFP_KERNEL);
+		m->buf = kmalloc_large(m->size <<= 1, GFP_KERNEL);
 		if (!m->buf)
 			goto Enomem;
 		m->count = 0;

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Hi Christoph,

Thanks for the point, do you plan to make kmalloc_large available for extern access in a separate mainline patch? Since kmalloc_large is statically defined in slub_def.h and when including it to seq_file.c
we have a lot of conflicting types:
..
In file included from ../linux/fs/seq_file.c:8:0:
../linux/include/linux/slub_def.h: In function 'kmalloc':
../linux/include/linux/slub_def.h:161:14: error: 'KMALLOC_MAX_CACHE_SIZE' undeclared (first use in this function) ../results/linux/include/linux/slub_def.h:161:14: note: each undeclared identifier is reported only once for each function it appears in ../linux/include/linux/slub_def.h:165:4: error: implicit declaration of function 'kmalloc_index' [-Werror=implicit-function-declaration] ../linux/include/linux/slub_def.h:168:12: error: 'ZERO_SIZE_PTR' undeclared (first use in this function) ../linux/include/linux/slub_def.h:170:34: error: 'kmalloc_caches' undeclared (first use in this function)
..

Thanks & BR
Wladislav Wiebe

You cannot separatly include slub_def.h. slab.h includes slub_def.h for you. What problem did you try to fix by doing so?

There is a patch pending that moves kmalloc_large to slab.h. So maybe we have to wait a merge period in order to be able to use it with other allocators than slub.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Hi,

ok, just saw in slab/for-linus branch that those stuff is reverted again..

commit 4932163637fbb9aaa654ca0703c5a624b7809da2

Author: Pekka Enberg <[email protected]>
Date:   Wed Jul 10 10:16:01 2013 +0300

Revert "mm/sl[aou]b: Move kmalloc_node functions to common code"

..

commit 35be03cafb8f5ddcc1236e90144b6ec76296b789

Author: Pekka Enberg <[email protected]>
Date:   Wed Jul 10 09:56:49 2013 +0300

Revert "mm/sl[aou]b: Move kmalloc definitions to slab.h"

--
WBR, WLadislav Wiebe

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

No that was only for the 3.11 merge by Linus. The 3.12 patches have not been put into pekkas tree.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Hi,

[...]

It seems some procfs file is trying to dump 8 MB at a single go. You need to fix that to return data in smaller chunks. What file is it?

A.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Hi,

seems it's coming from DT:
..
DEBUG: xxx seq_path: file path = /proc/device-tree/localbus@5000/flash@0/partition@1/reg

------------[ cut here ]------------
..

need to check if and how it's possible divide it in smaller chunks. If somebody has suggestions, feel free to comment-)

Thanks & BR
Wladislav Wiebe

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Hi guys,

we got the real root cause of the allocation issue:

Subject: [PATCH 1/1] of: fdt: fix memory initialization for expanded DT

Already existing property flags are filled wrong for properties created from initial FDT. This could cause problems if this DYNAMIC device-tree functions are used later, i.e. properties are attached/detached/replaced. Simply dumping flags from the running system show, that some initial static (not allocated via kzmalloc()) nodes are marked as dynamic.

I putted some debug extensions to property_proc_show(..) :
..
+ if (OF_IS_DYNAMIC(pp))
+ pr_err("DEBUG: xxx : OF_IS_DYNAMIC\n");
+ if (OF_IS_DETACHED(pp))
+ pr_err("DEBUG: xxx : OF_IS_DETACHED\n");

when you operate on the nodes (e.g.: ~$ cat /proc/device-tree/*some_node*) you will see that those flags are filled wrong, basically in most cases it will dump a DYNAMIC or DETACHED status, which is in not true.
(BTW. this OF_IS_DETACHED is a own define for debug purposes which which just make a test_bit(OF_DETACHED, &x->_flags)

If nodes are dynamic kernel is allowed to kfree() them. But it will crash attempting to do so on the nodes from FDT -- they are not allocated via kzmalloc().

---
 drivers/of/fdt.c |    2 ++
 1 files changed, 2 insertions(+), 0 deletions(-)

index 6bb7cf2..b10ba00 100644
@@ -392,6 +392,8 @@ static void __unflatten_device_tree(struct boot_param_header *blob,
 	mem = (unsigned long)
 		dt_alloc(size + 4, __alignof__(struct device_node));
 
+	memset((void *)mem, 0, size);
+
 	((__be32 *)mem)[size / 4] = cpu_to_be32(0xdeadbeef);
 
 	pr_debug("  unflattening %lx...\n", mem);

-- 1.7.1

This is committed to the mainline - hope it comes in soon.

Thanks & BR,
Wladislav Wiebe

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

This should go into stable as well...

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>