Skip to contentSkip to conversation

INFO: possible circular locking dependency detected

DISCUSSION alpha @ steudten Engineering 4 participants 6 messages Oct 16, 2006 – Oct 19, 2006

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

=======================================================
[ INFO: possible circular locking dependency detected ] 2.6.18-1.2189self #1
-------------------------------------------------------
kswapd0/186 is trying to acquire lock:
(&inode->i_mutex){--..}, at: [<c0326e32>] mutex_lock+0x21/0x24

but task is already holding lock:
(iprune_mutex){--..}, at: [<c0326e32>] mutex_lock+0x21/0x24

which lock already depends on the new lock.

the existing dependency chain (in reverse order) is:

-> #1 (iprune_mutex){--..}:

       [<c012fe17>] lock_acquire+0x4b/0x6c
       [<c0326cc4>] __mutex_lock_slowpath+0xb3/0x200
       [<c0326e32>] mutex_lock+0x21/0x24
       [<c017638a>] shrink_icache_memory+0x36/0x1b2
       [<c014cf50>] shrink_slab+0xd0/0x123
       [<c014d70e>] try_to_free_pages+0x114/0x1ba
       [<c0149b78>] __alloc_pages+0x18b/0x279
       [<c0146b10>] generic_file_buffered_write+0x167/0x546
       [<c014722f>] __generic_file_aio_write_nolock+0x340/0x38a
       [<c01472d5>] generic_file_aio_write+0x5c/0xaf
       [<f9c9e1cf>] nfs_file_write+0x7b/0x97 [nfs]
       [<c015f976>] do_sync_write+0xaf/0xe4
       [<c0160229>] vfs_write+0xab/0x157
       [<c0160760>] sys_write+0x3b/0x60
       [<c0102df3>] syscall_call+0x7/0xb

-> #0 (&inode->i_mutex){--..}:

       [<c012fe17>] lock_acquire+0x4b/0x6c
       [<c0326cc4>] __mutex_lock_slowpath+0xb3/0x200
       [<c0326e32>] mutex_lock+0x21/0x24
       [<f921706d>] ntfs_put_inode+0x3d/0x75 [ntfs]
       [<c0175acd>] iput+0x33/0x6a
       [<f9216eaa>] ntfs_clear_big_inode+0x99/0xb2 [ntfs]
       [<c0175fdc>] clear_inode+0xd8/0x129
       [<c01762cf>] dispose_list+0x4c/0xd1
       [<c01764de>] shrink_icache_memory+0x18a/0x1b2
       [<c014cf50>] shrink_slab+0xd0/0x123
       [<c014d2a3>] kswapd+0x260/0x336
       [<c012a4b7>] kthread+0xb0/0xdd
       [<c0101005>] kernel_thread_helper+0x5/0xb

other info that might help us debug this:
2 locks held by kswapd0/186:
#0: (shrinker_rwsem){----}, at: [<c014cea5>] shrink_slab+0x25/0x123 #1: (iprune_mutex){--..}, at: [<c0326e32>] mutex_lock+0x21/0x24

stack backtrace:
 [<c0103fe1>] show_trace+0xd/0x10
 [<c010447d>] dump_stack+0x19/0x1b
 [<c012ef1e>] print_circular_bug_tail+0x59/0x64
 [<c012f70b>] __lock_acquire+0x7e2/0x986
 [<c012fe17>] lock_acquire+0x4b/0x6c
 [<c0326cc4>] __mutex_lock_slowpath+0xb3/0x200
 [<c0326e32>] mutex_lock+0x21/0x24
 [<f921706d>] ntfs_put_inode+0x3d/0x75 [ntfs]
 [<c0175acd>] iput+0x33/0x6a
 [<f9216eaa>] ntfs_clear_big_inode+0x99/0xb2 [ntfs]
 [<c0175fdc>] clear_inode+0xd8/0x129
 [<c01762cf>] dispose_list+0x4c/0xd1
 [<c01764de>] shrink_icache_memory+0x18a/0x1b2
 [<c014cf50>] shrink_slab+0xd0/0x123
 [<c014d2a3>] kswapd+0x260/0x336
 [<c012a4b7>] kthread+0xb0/0xdd
 [<c0101005>] kernel_thread_helper+0x5/0xb
#1 lore raw
Quoted 11 lines · alpha @ steudten Engineering wrote:

=======================================================
[ INFO: possible circular locking dependency detected ] 2.6.18-1.2189self #1
-------------------------------------------------------
kswapd0/186 is trying to acquire lock:
(&inode->i_mutex){--..}, at: [<c0326e32>] mutex_lock+0x21/0x24

but task is already holding lock:
(iprune_mutex){--..}, at: [<c0326e32>] mutex_lock+0x21/0x24

which lock already depends on the new lock.

Thanks. __grab_cache_page wants to clear __GFP_FS, because it is holding the i_mutex so we don't want to reenter the filesystem in page reclaim.

This would be an easy two liner, except those funny page_cache_alloc routines which take a mapping rather than a gfp_t argument :P

Anyway, I'll get around to writing the real patch and queue it up with my other buffered write deadlock fixes. It should be fairly unlikely to cause you a deadlock. You could give this quick patch a try, though. Does it fix your problem?

Signature · 1 line
SUSE Labs, Novell Inc.
include/linux/pagemap.h +1 -1
Index: linux-2.6/include/linux/pagemap.h
===================================================================
@@ -57,7 +57,7 @@ extern struct page *page_cache_alloc_col
 #else
 static inline struct page *page_cache_alloc(struct address_space *x)
 {
-	return alloc_pages(mapping_gfp_mask(x), 0);
+	return alloc_pages(mapping_gfp_mask(x)&~__GFP_FS, 0);
 }
 
 static inline struct page *page_cache_alloc_cold(struct address_space *x)
mm/filemap.c +3 -3
Index: linux-2.6/mm/filemap.c
===================================================================
@@ -471,9 +471,9 @@ struct page *page_cache_alloc(struct add
 {
 	if (cpuset_do_page_mem_spread()) {
 		int n = cpuset_mem_spread_node();
-		return alloc_pages_node(n, mapping_gfp_mask(x), 0);
+		return alloc_pages_node(n, mapping_gfp_mask(x)&~__GFP_FS, 0);
 	}
-	return alloc_pages(mapping_gfp_mask(x), 0);
+	return alloc_pages(mapping_gfp_mask(x)&~__GFP_FS, 0);
 }
 EXPORT_SYMBOL(page_cache_alloc);
 
@@ -1864,7 +1864,7 @@ repeat:
 				return NULL;
 		}
 		err = add_to_page_cache(*cached_page, mapping,
-							index, GFP_KERNEL);
+						index, GFP_KERNEL&~__GFP_FS);
 		if (err == -EEXIST)
 			goto repeat;
 		if (err == 0) {
#2 lore raw
Quoted 11 lines · On Tue, 17 Oct 2006 00:32:44 +1000 Nick Piggin wrote:
Quoted 10 lines · alpha @ steudten Engineering wrote:

=======================================================
[ INFO: possible circular locking dependency detected ] 2.6.18-1.2189self #1
-------------------------------------------------------
kswapd0/186 is trying to acquire lock:
(&inode->i_mutex){--..}, at: [<c0326e32>] mutex_lock+0x21/0x24

but task is already holding lock:
(iprune_mutex){--..}, at: [<c0326e32>] mutex_lock+0x21/0x24

Quoted 8 lines
Quoted 1 line

which lock already depends on the new lock.

Thanks. __grab_cache_page wants to clear __GFP_FS, because it is holding the i_mutex so we don't want to reenter the filesystem in page reclaim.

This would be an easy two liner, except those funny page_cache_alloc routines which take a mapping rather than a gfp_t argument :P

and it would be only one email, but you forgot spaces there, so it's too ugly to use. ;) i.e., please add spaces around the '&'.

and it's an attachment :(

Quoted 4 lines

Anyway, I'll get around to writing the real patch and queue it up with my other buffered write deadlock fixes. It should be fairly unlikely to cause you a deadlock. You could give this quick patch a try, though. Does it fix your problem?

---
~Randy

#3 lore raw
Quoted 25 lines · Randy Dunlap
Quoted 12 lines · On Tue, 17 Oct 2006 00:32:44 +1000 Nick Piggin wrote:
Quoted 10 lines · alpha @ steudten Engineering wrote:

=======================================================
[ INFO: possible circular locking dependency detected ] 2.6.18-1.2189self #1
-------------------------------------------------------
kswapd0/186 is trying to acquire lock:
(&inode->i_mutex){--..}, at: [<c0326e32>] mutex_lock+0x21/0x24

but task is already holding lock:
(iprune_mutex){--..}, at: [<c0326e32>] mutex_lock+0x21/0x24

Quoted 8 lines
Quoted 1 line

which lock already depends on the new lock.

Thanks. __grab_cache_page wants to clear __GFP_FS, because it is holding the i_mutex so we don't want to reenter the filesystem in page reclaim.

This would be an easy two liner, except those funny page_cache_alloc routines which take a mapping rather than a gfp_t argument :P

Quoted 5 lines

and it would be only one email, but you forgot spaces there, so it's too ugly to use. ;) i.e., please add spaces around the '&'.

and it's an attachment :(

Oh yeah, it was an attachment because I didn't want anyone to see how ugly it is ;)

It's just a quick hack to see if it works. I'll send out a real patch when I get time to code and test it properly.

Signature · 2 lines
SUSE Labs, Novell Inc.
Send instant messages to your online friends http://au.messenger.yahoo.com 
#4 lore raw
In reply to Nick Piggin
Quoted 8 lines · On Tue, 17 Oct 2006 00:32:44 +1000 Nick Piggin <[email protected]> wrote:
Quoted 7 lines · alpha @ steudten Engineering wrote:

=======================================================
[ INFO: possible circular locking dependency detected ] 2.6.18-1.2189self #1
-------------------------------------------------------
kswapd0/186 is trying to acquire lock:
(&inode->i_mutex){--..}, at: [<c0326e32>] mutex_lock+0x21/0x24

Quoted 8 lines
Quoted 4 lines

but task is already holding lock:
(iprune_mutex){--..}, at: [<c0326e32>] mutex_lock+0x21/0x24

which lock already depends on the new lock.

Thanks. __grab_cache_page wants to clear __GFP_FS, because it is holding the i_mutex so we don't want to reenter the filesystem in page reclaim.

We want to be able to enter page reclaim while holding i_mutex. Think what the effect of not doing this would be upon write() (!)

This warning is more fallout from ntfs's insistence on taking i_mutex in its clear_inode(). See lengthy and unproductive discussion at
http://lkml.org/lkml/2006/7/26/185 .

#5 lore raw
Quoted 23 lines · Andrew Morton
Quoted 9 lines · On Tue, 17 Oct 2006 00:32:44 +1000 Nick Piggin <[email protected]> wrote:
Quoted 7 lines · alpha @ steudten Engineering wrote:

=======================================================
[ INFO: possible circular locking dependency detected ] 2.6.18-1.2189self #1
-------------------------------------------------------
kswapd0/186 is trying to acquire lock:
(&inode->i_mutex){--..}, at: [<c0326e32>] mutex_lock+0x21/0x24

Quoted 8 lines
Quoted 4 lines

but task is already holding lock:
(iprune_mutex){--..}, at: [<c0326e32>] mutex_lock+0x21/0x24

which lock already depends on the new lock.

Thanks. __grab_cache_page wants to clear __GFP_FS, because it is holding the i_mutex so we don't want to reenter the filesystem in page reclaim.

Quoted 6 lines

We want to be able to enter page reclaim while holding i_mutex. Think what the effect of not doing this would be upon write() (!)

This warning is more fallout from ntfs's insistence on taking i_mutex in its clear_inode(). See lengthy and unproductive discussion at
http://lkml.org/lkml/2006/7/26/185 .

Yeah you're right. It will be a hot allocation + reclaim path for high bandwidth writes.

Signature · 2 lines
SUSE Labs, Novell Inc.
Send instant messages to your online friends http://au.messenger.yahoo.com