2.6.23-rc4-mm1 memory controller BUG_ON()

Looks like somebody is holding a lock while trying to do a mem_container_charge(), and the mem_container_charge() call is doing an allocation. Naughty.

I'm digging into it a bit more, but thought I'd report it, first.

.config: http://sr71.net/~dave/linux/memory-controller-bug.config

BUG: sleeping function called from invalid context at /home/dave/work/linux/2.6/23/rc4/mm1/lxc/mm/slab.c:3052
in_atomic():1, irqs_disabled():0
 [<c01029c1>] show_trace_log_lvl+0x19/0x2e
 [<c01029e8>] show_trace+0x12/0x14
 [<c0102ad3>] dump_stack+0x13/0x15
 [<c010f223>] __might_sleep+0xe4/0xea
 [<c014cdc0>] kmem_cache_alloc+0x25/0xae
 [<c014e1dd>] mem_container_charge+0xc9/0x2cd
 [<c014e403>] mem_container_cache_charge+0x22/0x28
 [<c0131839>] add_to_page_cache+0x35/0xd7
 [<c01318f0>] add_to_page_cache_lru+0x15/0x29
 [<c0131c43>] find_or_create_page+0x75/0x93
 [<c016cc71>] grow_dev_page+0x32/0x125
 [<c016ce15>] grow_buffers+0xb1/0xd4
 [<c016ceef>] __getblk_slow+0xb7/0xcf
 [<c016d2b9>] __getblk+0x44/0x4f
 [<c018c721>] ext3_getblk+0xca/0x19c
 [<c018fc48>] ext3_find_entry+0x127/0x325
 [<c0190063>] ext3_lookup+0x2c/0xe1
 [<c015635a>] real_lookup+0x54/0xc5
 [<c01565bd>] do_lookup+0x59/0xa0
 [<c0156824>] __link_path_walk+0x220/0xa4f
 [<c0157094>] link_path_walk+0x41/0xa5
 [<c0157110>] path_walk+0x18/0x1a
 [<c01573de>] do_path_lookup+0x165/0x182
 [<c01574a0>] __path_lookup_intent_open+0x44/0x75
 [<c01574f2>] path_lookup_open+0x21/0x27
 [<c0157c5a>] open_namei+0x7f/0x4c4
 [<c014f6c0>] do_filp_open+0x26/0x3b
 [<c014f975>] do_sys_open+0x43/0xc7
 [<c014fa13>] sys_open+0x1a/0x1c
 [<c010010e>] init_post+0x45/0xe7
 [<c03f6ab6>] kernel_init+0x8a/0x8e
 [<c010287b>] kernel_thread_helper+0x7/0x10
 =======================

-- Dave

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

I'm now thinking this is because the add_to_page_cache() functions have a gfp_mask passed in, and the mem_container_charge() functions don't take that mask. So, even if the add_to_page_cache() user specified ! __GFP_WAIT, the mem_container_charge() function can sleep on its kmalloc.

I'll try passing gfp_flags through to it and see what happens.

-- Dave

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Please see my patch at

http://lkml.org/lkml/2007/9/12/113 and some more details as a reply to your earlier email.

	Warm Regards,
	Balbir Singh
	Linux Technology Center
	IBM, ISTL

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Hi, Dave,

Thanks for reporting this. I sent out a patch to fix this problem (suggested by Nick Piggin). The patch is available at

http://lkml.org/lkml/2007/9/12/113

Could you try the patch and check if the problem goes away?

Any pointers on how to reproduce the problem would be extremely useful.

	Warm Regards,
	Balbir Singh
	Linux Technology Center
	IBM, ISTL

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Balbir and I had a chat about this on IRC. Those patches don't seem to fix it. But, I'm getting Balbir hooked up with the kvm instance that I ran this in along with my .config.

-- Dave

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Hi, Dave,

Here's a fix for the problem, it works nicely on the qemu setup that you helped me with. I am also testing on another box. Could you please verify if the patch helps?

Description of the patch
------------------------

Move mem_controller_cache_charge() above radix_tree_preload(). radix_tree_preload() disables preemption, even though the gfp_mask passed contains __GFP_WAIT, we cannot really do __GFP_WAIT allocations, thus we hit a BUG_ON() in kmem_cache_alloc().

This patch moves mem_controller_cache_charge() to above radix_tree_preload() for cache charging.

---

 mm/filemap.c    |   13 ++++++-------
 mm/swap_state.c |   13 +++++++------
 2 files changed, 13 insertions(+), 13 deletions(-)

diff -puN mm/filemap.c~memory-controller-make-charging-gfpmask-aware-fixes mm/filemap.c
@@ -441,14 +441,12 @@ int filemap_write_and_wait_range(struct 
 int add_to_page_cache(struct page *page, struct address_space *mapping,
 		pgoff_t offset, gfp_t gfp_mask)
 {
-	int error = radix_tree_preload(gfp_mask & ~__GFP_HIGHMEM);
+	int error = mem_container_cache_charge(page, current->mm, gfp_mask);
+	if (error)
+		goto out;
 
+	error = radix_tree_preload(gfp_mask & ~__GFP_HIGHMEM);
 	if (error == 0) {
-
-		error = mem_container_cache_charge(page, current->mm, gfp_mask);
-		if (error)
-			goto out;
-
 		write_lock_irq(&mapping->tree_lock);
 		error = radix_tree_insert(&mapping->page_tree, offset, page);
 		if (!error) {

@@ -463,7 +461,8 @@ int add_to_page_cache(struct page *page,
 
 		write_unlock_irq(&mapping->tree_lock);
 		radix_tree_preload_end();
-	}
+	} else
+		mem_container_uncharge_page(page);
 out:
 	return error;
 }

diff -puN mm/swap_state.c~memory-controller-make-charging-gfpmask-aware-fixes mm/swap_state.c
@@ -78,13 +78,13 @@ static int __add_to_swap_cache(struct pa
 	BUG_ON(!PageLocked(page));
 	BUG_ON(PageSwapCache(page));
 	BUG_ON(PagePrivate(page));
-	error = radix_tree_preload(gfp_mask);
-	if (!error) {
 
-		error = mem_container_cache_charge(page, current->mm, gfp_mask);
-		if (error)
-			goto out;
+	error = mem_container_cache_charge(page, current->mm, gfp_mask);
+	if (error)
+		goto out;
 
+	error = radix_tree_preload(gfp_mask);
+	if (!error) {
 		write_lock_irq(&swapper_space.tree_lock);
 		error = radix_tree_insert(&swapper_space.page_tree,
 						entry.val, page);

@@ -99,7 +99,8 @@ static int __add_to_swap_cache(struct pa
 
 		write_unlock_irq(&swapper_space.tree_lock);
 		radix_tree_preload_end();
-	}
+	} else
+		mem_container_uncharge_page(page);
 out:
 	return error;
 }

_

	Warm Regards,
	Balbir Singh
	Linux Technology Center
	IBM, ISTL

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>