Skip to contentSkip to conversation

drop_pagecache: Possible circular locking dependency

DISCUSSION Fengguang Wu 3 participants 5 messages Dec 4, 2006

Subsystems (6)
lore
Share & tools
lore
#0 lore raw

Hi Ingo,

Got the following message when doing some benchmarks.
I guess we should not hold inode_lock on calling invalidate_inode_pages(). Any ideas?

Fengguang Wu

=======================================================
[ INFO: possible circular locking dependency detected ]
2.6.19-rc6-mm2 #3
-------------------------------------------------------
rabench.sh/7467 is trying to acquire lock:
(&journal->j_list_lock){--..}, at: [<ffffffff8113bdbc>] journal_try_to_free_buffers+0xdc/0x1c0

but task is already holding lock:
(inode_lock){--..}, at: [<ffffffff810fe857>] drop_pagecache+0x67/0x120

which lock already depends on the new lock.

the existing dependency chain (in reverse order) is:

-> #1 (inode_lock){--..}:

       [<ffffffff810b098f>] add_lock_to_list+0x8f/0xd0
       [<ffffffff810b35b9>] __lock_acquire+0xb29/0xd40
       [<ffffffff810b3863>] lock_acquire+0x93/0xc0
       [<ffffffff81072875>] _spin_lock+0x25/0x40
       [<ffffffff81014e41>] __mark_inode_dirty+0x101/0x190
       [<ffffffff8100fa00>] __set_page_dirty_nobuffers+0x150/0x170
       [<ffffffff810314ae>] mark_buffer_dirty+0x1e/0x30
       [<ffffffff81139798>] __journal_temp_unlink_buffer+0x1a8/0x1c0
       [<ffffffff81139a31>] __journal_unfile_buffer+0x11/0x20
       [<ffffffff81139ac7>] __journal_refile_buffer+0x87/0x120
       [<ffffffff8113cfe5>] journal_commit_transaction+0x1005/0x1300
       [<ffffffff81141285>] kjournald+0xd5/0x230
       [<ffffffff810393ea>] kthread+0xda/0x110
       [<ffffffff8106bf38>] child_rip+0xa/0x12
       [<ffffffffffffffff>] 0xffffffffffffffff

-> #0 (&journal->j_list_lock){--..}:

       [<ffffffff810b19e5>] print_circular_bug_tail+0x55/0xb0
       [<ffffffff810b349e>] __lock_acquire+0xa0e/0xd40
       [<ffffffff810b3863>] lock_acquire+0x93/0xc0
       [<ffffffff81072875>] _spin_lock+0x25/0x40
       [<ffffffff8113bdbc>] journal_try_to_free_buffers+0xdc/0x1c0
       [<ffffffff8112af49>] ext3_releasepage+0xa9/0xd0
       [<ffffffff8102c1ba>] try_to_release_page+0x5a/0x80
       [<ffffffff8104f438>] invalidate_mapping_pages+0xa8/0x150
       [<ffffffff81066842>] invalidate_inode_pages+0x12/0x20
       [<ffffffff810fe895>] drop_pagecache+0xa5/0x120
       [<ffffffff810fe932>] drop_caches_sysctl_handler+0x22/0x80
       [<ffffffff8109dca7>] do_rw_proc+0xe7/0x150
       [<ffffffff8109dd2a>] proc_writesys+0x1a/0x20
       [<ffffffff81018344>] vfs_write+0xf4/0x1b0
       [<ffffffff81018ea0>] sys_write+0x50/0xa0
       [<ffffffff8106b11e>] system_call+0x7e/0x83
       [<00002af654962422>] 0x2af654962422
       [<ffffffffffffffff>] 0xffffffffffffffff

other info that might help us debug this:

2 locks held by rabench.sh/7467:
#0: (&type->s_umount_key#16){----}, at: [<ffffffff810fe843>] drop_pagecache+0x53/0x120 #1: (inode_lock){--..}, at: [<ffffffff810fe857>] drop_pagecache+0x67/0x120

stack backtrace:

Call Trace:
 [<ffffffff8107a3e3>] dump_trace+0xb3/0x460
 [<ffffffff8107a7d3>] show_trace+0x43/0x70
 [<ffffffff8107a815>] dump_stack+0x15/0x20
 [<ffffffff810b1a26>] print_circular_bug_tail+0x96/0xb0
 [<ffffffff810b349e>] __lock_acquire+0xa0e/0xd40
 [<ffffffff810b3863>] lock_acquire+0x93/0xc0
 [<ffffffff81072875>] _spin_lock+0x25/0x40
 [<ffffffff8113bdbc>] journal_try_to_free_buffers+0xdc/0x1c0
 [<ffffffff8112af49>] ext3_releasepage+0xa9/0xd0
 [<ffffffff8102c1ba>] try_to_release_page+0x5a/0x80
 [<ffffffff8104f438>] invalidate_mapping_pages+0xa8/0x150
 [<ffffffff81066842>] invalidate_inode_pages+0x12/0x20
 [<ffffffff810fe895>] drop_pagecache+0xa5/0x120
 [<ffffffff810fe932>] drop_caches_sysctl_handler+0x22/0x80
 [<ffffffff8109dca7>] do_rw_proc+0xe7/0x150
 [<ffffffff8109dd2a>] proc_writesys+0x1a/0x20
 [<ffffffff81018344>] vfs_write+0xf4/0x1b0
 [<ffffffff81018ea0>] sys_write+0x50/0xa0
 [<ffffffff8106b11e>] system_call+0x7e/0x83
 [<00002af654962422>]

2 locks held by rabench.sh/7467:
#0: (&type->s_umount_key#16){----}, at: [<ffffffff810fe843>] drop_pagecache+0x53/0x120 #1: (inode_lock){--..}, at: [<ffffffff810fe857>] drop_pagecache+0x67/0x120

#1 lore raw
Quoted 13 lines · On Mon, 4 Dec 2006 16:09:02 +0800 Fengguang Wu <[email protected]> wrote:

Got the following message when doing some benchmarks.
I guess we should not hold inode_lock on calling invalidate_inode_pages(). Any ideas?

Fengguang Wu

=======================================================
[ INFO: possible circular locking dependency detected ]
2.6.19-rc6-mm2 #3
-------------------------------------------------------
rabench.sh/7467 is trying to acquire lock:
(&journal->j_list_lock){--..}, at: [<ffffffff8113bdbc>] journal_try_to_free_buffers+0xdc/0x1c0

Quoted 5 lines

but task is already holding lock:
(inode_lock){--..}, at: [<ffffffff810fe857>] drop_pagecache+0x67/0x120

which lock already depends on the new lock.

drat, I was afraid someone would notice.

It's Hard To Fix. Removing /proc/sys/vm/drop_pagecache would in fact be my preferred fix.

#2 lore raw
Quoted 19 lines · Andrew Morton
Quoted 16 lines · On Mon, 4 Dec 2006 16:09:02 +0800 Fengguang Wu <[email protected]> wrote:

Got the following message when doing some benchmarks.
I guess we should not hold inode_lock on calling invalidate_inode_pages(). Any ideas?

Fengguang Wu

=======================================================
[ INFO: possible circular locking dependency detected ]
2.6.19-rc6-mm2 #3
-------------------------------------------------------
rabench.sh/7467 is trying to acquire lock:
(&journal->j_list_lock){--..}, at: [<ffffffff8113bdbc>] journal_try_to_free_buffers+0xdc/0x1c0

but task is already holding lock:
(inode_lock){--..}, at: [<ffffffff810fe857>] drop_pagecache+0x67/0x120

Quoted 7 lines
Quoted 2 lines

which lock already depends on the new lock.

drat, I was afraid someone would notice.

It's Hard To Fix. Removing /proc/sys/vm/drop_pagecache would in fact be my preferred fix.

Or fix drop_pagecache_sb():
Repeat until all possible pages freed:
grab and save some inodes to a buffer zip their pages outside of inode_lock

Takes much more code though.

I'd like to move this sysctl interface to the upcoming /proc/filecache. Being a module, it helps reduce the kernel size :)

Fengguang Wu

#3 lore raw
Quoted 1 line · Fengguang Wu

I'd like to move this sysctl interface to the upcoming /proc/filecache. Being a module, it helps reduce the kernel size :)

What's /proc/filecache?

Signature · 2 lines
SUSE Labs, Novell Inc.
Send instant messages to your online friends http://au.messenger.yahoo.com 
#4 lore raw
Quoted 6 lines · Nick Piggin
Quoted 2 lines · Fengguang Wu

I'd like to move this sysctl interface to the upcoming /proc/filecache. Being a module, it helps reduce the kernel size :)

What's /proc/filecache?

It provides access to the in-memory file cache.

To list an index of all cached files:

echo -n index > /proc/filecache
cat /proc/filecache

The output looks like:

    # filecache 1.0
    #      ino       size   cached cached%  state   refcnt  dev             file
       1026334         91       92    100   --      66      03:02(hda2)     /lib/ld-2.3.6.so
        233608       1242      972     78   --      66      03:02(hda2)     /lib/tls/libc-2.3.6.so
         65203        651      476     73   --      1       03:02(hda2)     /bin/bash
       1026445        261      160     61   --      10      03:02(hda2)     /lib/libncurses.so.5.5
        235427         10       12    100   --      44      03:02(hda2)     /lib/tls/libdl-2.3.6.so

To list the cached pages of a perticular file:

echo -n /bin/bash > /proc/filecache
cat /proc/filecache

# file /bin/bash
# flags R:referenced A:active U:uptodate D:dirty W:writeback M:mmap
# idx len state refcnt
0 36 RAU__M 3
36 1 RAU__M 2
37 8 RAU__M 3
45 2 RAU___ 1
47 6 RAU__M 3
53 3 RAU__M 2
56 2 RAU__M 3

To instruct the kernel to drop clean caches, dentries and inodes from memory, causing that memory to become free:

    # drop clean file data cache (i.e. file backed pagecache)
    echo drop data > /proc/filecache
    # drop clean file metadata cache (i.e. dentries and inodes)
    echo drop metadata > /proc/filecache

It's first introduced here: http://lkml.org/lkml/2006/6/12/29

And the latest working patch for -mm:

include/linux/mm.h +1 -0
@@ -29,6 +29,7 @@ extern unsigned long num_physpages;
 extern void * high_memory;
 extern unsigned long vmalloc_earlyreserve;
 extern int page_cluster;
+extern char *zone_names[];
 
 #ifdef CONFIG_SYSCTL
 extern int sysctl_legacy_va_layout;
mm/page_alloc.c +1 -1
@@ -86,7 +86,7 @@ int sysctl_lowmem_reserve_ratio[MAX_NR_Z
 
 EXPORT_SYMBOL(totalram_pages);
 
-static char *zone_names[MAX_NR_ZONES] = {
+char *zone_names[MAX_NR_ZONES] = {
 #ifdef CONFIG_ZONE_DMA
 	 "DMA",
 #endif
fs/dcache.c +4 -1
@@ -1776,7 +1776,10 @@ static char * __d_path( struct dentry *d
 
 		if (dentry == root && vfsmnt == rootmnt)
 			break;
-		if (dentry == vfsmnt->mnt_root || IS_ROOT(dentry)) {
+		if (unlikely(!vfsmnt)) {
+			if (IS_ROOT(dentry))
+				break;
+		} else if (dentry == vfsmnt->mnt_root || IS_ROOT(dentry)) {
 			/* Global root? */
 			spin_lock(&vfsmount_lock);
 			if (vfsmnt->mnt_parent == vfsmnt) {
fs/seq_file.c +12 -9
@@ -13,6 +13,8 @@
 #include <asm/uaccess.h>
 #include <asm/page.h>
 
+#define SEQFILE_SHOW_FROM_NEXT	LONG_MAX
+
 /**
  *	seq_open -	initialize sequential file
  *	@file: file we initialize
@@ -93,6 +95,7 @@ ssize_t seq_read(struct file *file, char
 	/* if not empty - flush it first */
 	if (m->count) {
 		n = min(m->count, size);
+		BUG_ON(m->from == SEQFILE_SHOW_FROM_NEXT);
 		err = copy_to_user(buf, m->buf + m->from, n);
 		if (err)
 			goto Efault;
@@ -102,7 +105,7 @@ ssize_t seq_read(struct file *file, char
 		buf += n;
 		copied += n;
 		if (!m->count)
-			m->index++;
+			m->from = SEQFILE_SHOW_FROM_NEXT;
 		if (!size)
 			goto Done;
 	}
@@ -113,9 +116,11 @@ ssize_t seq_read(struct file *file, char
 		err = PTR_ERR(p);
 		if (!p || IS_ERR(p))
 			break;
-		err = m->op->show(m, p);
-		if (err)
-			break;
+		if (m->from != SEQFILE_SHOW_FROM_NEXT) {
+			err = m->op->show(m, p);
+			if (err)
+				break;
+		}
 		if (m->count < m->size)
 			goto Fill;
 		m->op->stop(m, p);
@@ -156,7 +161,7 @@ Fill:
 	if (m->count)
 		m->from = n;
 	else
-		pos++;
+		m->from = SEQFILE_SHOW_FROM_NEXT;
 	m->index = pos;
 Done:
 	if (!copied)
@@ -208,11 +213,9 @@ static int traverse(struct seq_file *m, 
 		}
 		pos += m->count;
 		m->count = 0;
-		if (pos == offset) {
-			m->index++;
-			break;
-		}
 		p = m->op->next(m, p, &m->index);
+		if (pos == offset)
+			break;
 	}
 	m->op->stop(m, p);
 	return error;
Documentation/filesystems/proc.txt +83 -0

95 lines not on this page · raw message

fs/proc/filecache.c new file +614 -0

614 lines not on this page · raw message

fs/Kconfig +23 -0
@@ -1047,6 +1047,29 @@ config CONFIGFS_FS
 	  Both sysfs and configfs can and should exist together on the
 	  same system. One is not a replacement for the other.
 
+config PROC_FILECACHE
+	tristate "/proc/filecache support"
+	default m
+	depends on PROC_FS
+	help
+	  This option creates a file /proc/filecache which enables one to
+	  query/drop the cached files in memory.
+
+	  A quick start guide:
+
+	  # echo -n index > /proc/filecache
+	  # cat /proc/filecache
+
+	  # echo -n /bin/bash > /proc/filecache
+	  # cat /proc/filecache
+
+	  # echo drop data > /proc/filecache
+	  # echo drop metadata > /proc/filecache
+
+	  For more details, please check Documentation/filesystems/proc.txt .
+
+	  It can be a handy tool for sysadms and desktop users.
+
 endmenu
 
 menu "Miscellaneous filesystems"
fs/proc/Makefile +1 -0
@@ -12,5 +12,6 @@ proc-y       += inode.o root.o base.o ge
 
 proc-$(CONFIG_PROC_KCORE)	+= kcore.o
 proc-$(CONFIG_PROC_VMCORE)	+= vmcore.o
+proc-$(CONFIG_PROC_FILECACHE)	+= filecache.o
 proc-$(CONFIG_PROC_DEVICETREE)	+= proc_devtree.o
 proc-$(CONFIG_PRINTK)	+= kmsg.o